Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................X...X......XX.............] duration: 5.855575843s trace: https://jaeger.dev.storj.io/trace/6f5c529e7f01e189 FAIL (2025-11-15 16:49:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 07s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079179359 (152h34m23.63064982s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079536990 (152h34m23.631007451s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079909870 (152h34m23.631380351s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080308579 (152h34m23.631779041s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080706457 (152h34m23.632176908s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081011267 (152h34m23.632481728s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081354641 (152h34m23.632825102s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081749092 (152h34m23.633219543s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.082106943 (152h34m23.633577394s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084095681 (152h34m23.635566142s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084488068 (152h34m23.635958529s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084874634 (152h34m23.636345095s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:26.855580766 (152h34m25.407051217s) [upload=134380]: Next returned nil segment Nov 15 16:38:26.855606955 (152h34m25.407077416s) [upload=134380]: Waiting for error group managing segments... Nov 15 16:38:27.202416630 (152h34m25.753887081s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.224498370 (152h34m25.775968831s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.239576622 (152h34m25.791047084s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.283074694 (152h34m25.834545145s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.294108625 (152h34m25.845579086s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:47.996973416 (152h34m46.548443877s) [noise=true , address=145.79.227.21:4473 , piece_id=P7FNCN2PW445CVOUC6JBBUQYEC6ESHALJR2I5CCJ7ADB2R33OJJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:52.282269494 (152h34m50.833739955s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:54.127175860 (152h34m52.678646311s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection Nov 15 16:38:54.127196390 (152h34m52.678666851s) [noise=true , address=145.79.227.28:4726 , piece_id=DSSAU37SBLAVNIU3A46JPOGCNBTBT5B4AG767N5DA4OFZZZA3Y2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:38:54.127411158 (152h34m52.678881629s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection Nov 15 16:38:54.127524273 (152h34m52.678994725s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection Nov 15 16:38:54.127571524 (152h34m52.679041985s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:38:54.127613895 (152h34m52.679084356s) [noise=true , address=145.79.227.15:4874 , piece_id=JFR52BOF4X3JKDG4W2PWPALSICPMXUFW4ECMMJP3ZJ2EOZLXWFPQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection Nov 15 16:38:54.127679660 (152h34m52.679150111s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection Nov 15 16:38:54.127718884 (152h34m52.679189346s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection Nov 15 16:38:54.128019055 (152h34m52.679489516s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection Nov 15 16:38:54.128048231 (152h34m52.679518692s) [seg_pos={0 0}, upload=134380]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 16:38:54.375644018 (152h34m52.927114469s) [upload=134380]: Done uploading segments... Nov 15 16:38:54.375693672 (152h34m52.927164133s) [upload=134380]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.855612453s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X........X..............................................] duration: 3.539768005s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:48:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.539797712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................X...X......XX............] duration: 6.171446453s trace: https://jaeger.dev.storj.io/trace/6f5c529e7f01e189 FAIL (2025-11-15 16:48:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 07s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079179359 (152h34m23.63064982s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079536990 (152h34m23.631007451s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079909870 (152h34m23.631380351s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080308579 (152h34m23.631779041s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080706457 (152h34m23.632176908s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081011267 (152h34m23.632481728s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081354641 (152h34m23.632825102s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081749092 (152h34m23.633219543s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.082106943 (152h34m23.633577394s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084095681 (152h34m23.635566142s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084488068 (152h34m23.635958529s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084874634 (152h34m23.636345095s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:26.855580766 (152h34m25.407051217s) [upload=134380]: Next returned nil segment Nov 15 16:38:26.855606955 (152h34m25.407077416s) [upload=134380]: Waiting for error group managing segments... Nov 15 16:38:27.202416630 (152h34m25.753887081s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.224498370 (152h34m25.775968831s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.239576622 (152h34m25.791047084s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.283074694 (152h34m25.834545145s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.294108625 (152h34m25.845579086s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:47.996973416 (152h34m46.548443877s) [noise=true , address=145.79.227.21:4473 , piece_id=P7FNCN2PW445CVOUC6JBBUQYEC6ESHALJR2I5CCJ7ADB2R33OJJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:52.282269494 (152h34m50.833739955s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:54.127175860 (152h34m52.678646311s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection Nov 15 16:38:54.127196390 (152h34m52.678666851s) [noise=true , address=145.79.227.28:4726 , piece_id=DSSAU37SBLAVNIU3A46JPOGCNBTBT5B4AG767N5DA4OFZZZA3Y2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:38:54.127411158 (152h34m52.678881629s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection Nov 15 16:38:54.127524273 (152h34m52.678994725s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection Nov 15 16:38:54.127571524 (152h34m52.679041985s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:38:54.127613895 (152h34m52.679084356s) [noise=true , address=145.79.227.15:4874 , piece_id=JFR52BOF4X3JKDG4W2PWPALSICPMXUFW4ECMMJP3ZJ2EOZLXWFPQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection Nov 15 16:38:54.127679660 (152h34m52.679150111s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection Nov 15 16:38:54.127718884 (152h34m52.679189346s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection Nov 15 16:38:54.128019055 (152h34m52.679489516s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection Nov 15 16:38:54.128048231 (152h34m52.679518692s) [seg_pos={0 0}, upload=134380]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 16:38:54.375644018 (152h34m52.927114469s) [upload=134380]: Done uploading segments... Nov 15 16:38:54.375693672 (152h34m52.927164133s) [upload=134380]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.171474105s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....X........X.............................................] duration: 3.539442078s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:47:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.539610209s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......X........X............................................] duration: 3.497718487s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:47:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.497780815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................X...X......XX...........] duration: 5.954229873s trace: https://jaeger.dev.storj.io/trace/6f5c529e7f01e189 FAIL (2025-11-15 16:47:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 07s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079179359 (152h34m23.63064982s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079536990 (152h34m23.631007451s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079909870 (152h34m23.631380351s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080308579 (152h34m23.631779041s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080706457 (152h34m23.632176908s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081011267 (152h34m23.632481728s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081354641 (152h34m23.632825102s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081749092 (152h34m23.633219543s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.082106943 (152h34m23.633577394s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084095681 (152h34m23.635566142s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084488068 (152h34m23.635958529s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084874634 (152h34m23.636345095s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:26.855580766 (152h34m25.407051217s) [upload=134380]: Next returned nil segment Nov 15 16:38:26.855606955 (152h34m25.407077416s) [upload=134380]: Waiting for error group managing segments... Nov 15 16:38:27.202416630 (152h34m25.753887081s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.224498370 (152h34m25.775968831s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.239576622 (152h34m25.791047084s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.283074694 (152h34m25.834545145s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.294108625 (152h34m25.845579086s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:47.996973416 (152h34m46.548443877s) [noise=true , address=145.79.227.21:4473 , piece_id=P7FNCN2PW445CVOUC6JBBUQYEC6ESHALJR2I5CCJ7ADB2R33OJJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:52.282269494 (152h34m50.833739955s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:54.127175860 (152h34m52.678646311s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection Nov 15 16:38:54.127196390 (152h34m52.678666851s) [noise=true , address=145.79.227.28:4726 , piece_id=DSSAU37SBLAVNIU3A46JPOGCNBTBT5B4AG767N5DA4OFZZZA3Y2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:38:54.127411158 (152h34m52.678881629s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection Nov 15 16:38:54.127524273 (152h34m52.678994725s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection Nov 15 16:38:54.127571524 (152h34m52.679041985s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:38:54.127613895 (152h34m52.679084356s) [noise=true , address=145.79.227.15:4874 , piece_id=JFR52BOF4X3JKDG4W2PWPALSICPMXUFW4ECMMJP3ZJ2EOZLXWFPQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection Nov 15 16:38:54.127679660 (152h34m52.679150111s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection Nov 15 16:38:54.127718884 (152h34m52.679189346s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection Nov 15 16:38:54.128019055 (152h34m52.679489516s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection Nov 15 16:38:54.128048231 (152h34m52.679518692s) [seg_pos={0 0}, upload=134380]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 16:38:54.375644018 (152h34m52.927114469s) [upload=134380]: Done uploading segments... Nov 15 16:38:54.375693672 (152h34m52.927164133s) [upload=134380]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 5.954262214s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......X........X...........................................] duration: 3.400980451s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:46:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.401007663s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................X...X......XX..........] duration: 12.626549293s trace: https://jaeger.dev.storj.io/trace/6f5c529e7f01e189 FAIL (2025-11-15 16:46:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 07s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079179359 (152h34m23.63064982s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079536990 (152h34m23.631007451s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079909870 (152h34m23.631380351s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080308579 (152h34m23.631779041s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080706457 (152h34m23.632176908s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081011267 (152h34m23.632481728s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081354641 (152h34m23.632825102s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081749092 (152h34m23.633219543s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.082106943 (152h34m23.633577394s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084095681 (152h34m23.635566142s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084488068 (152h34m23.635958529s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084874634 (152h34m23.636345095s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:26.855580766 (152h34m25.407051217s) [upload=134380]: Next returned nil segment Nov 15 16:38:26.855606955 (152h34m25.407077416s) [upload=134380]: Waiting for error group managing segments... Nov 15 16:38:27.202416630 (152h34m25.753887081s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.224498370 (152h34m25.775968831s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.239576622 (152h34m25.791047084s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.283074694 (152h34m25.834545145s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.294108625 (152h34m25.845579086s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:47.996973416 (152h34m46.548443877s) [noise=true , address=145.79.227.21:4473 , piece_id=P7FNCN2PW445CVOUC6JBBUQYEC6ESHALJR2I5CCJ7ADB2R33OJJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:52.282269494 (152h34m50.833739955s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:54.127175860 (152h34m52.678646311s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection Nov 15 16:38:54.127196390 (152h34m52.678666851s) [noise=true , address=145.79.227.28:4726 , piece_id=DSSAU37SBLAVNIU3A46JPOGCNBTBT5B4AG767N5DA4OFZZZA3Y2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:38:54.127411158 (152h34m52.678881629s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection Nov 15 16:38:54.127524273 (152h34m52.678994725s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection Nov 15 16:38:54.127571524 (152h34m52.679041985s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:38:54.127613895 (152h34m52.679084356s) [noise=true , address=145.79.227.15:4874 , piece_id=JFR52BOF4X3JKDG4W2PWPALSICPMXUFW4ECMMJP3ZJ2EOZLXWFPQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection Nov 15 16:38:54.127679660 (152h34m52.679150111s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection Nov 15 16:38:54.127718884 (152h34m52.679189346s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection Nov 15 16:38:54.128019055 (152h34m52.679489516s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection Nov 15 16:38:54.128048231 (152h34m52.679518692s) [seg_pos={0 0}, upload=134380]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 16:38:54.375644018 (152h34m52.927114469s) [upload=134380]: Done uploading segments... Nov 15 16:38:54.375693672 (152h34m52.927164133s) [upload=134380]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 12.626576134s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........X........X..........................................] duration: 3.438777328s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:45:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.438806003s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........X........X.........................................] duration: 3.543214054s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:45:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.543251054s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................X...X......XX.........] duration: 7.19411791s trace: https://jaeger.dev.storj.io/trace/6f5c529e7f01e189 FAIL (2025-11-15 16:44:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 07s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079179359 (152h34m23.63064982s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079536990 (152h34m23.631007451s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079909870 (152h34m23.631380351s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080308579 (152h34m23.631779041s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080706457 (152h34m23.632176908s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081011267 (152h34m23.632481728s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081354641 (152h34m23.632825102s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081749092 (152h34m23.633219543s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.082106943 (152h34m23.633577394s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084095681 (152h34m23.635566142s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084488068 (152h34m23.635958529s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084874634 (152h34m23.636345095s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:26.855580766 (152h34m25.407051217s) [upload=134380]: Next returned nil segment Nov 15 16:38:26.855606955 (152h34m25.407077416s) [upload=134380]: Waiting for error group managing segments... Nov 15 16:38:27.202416630 (152h34m25.753887081s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.224498370 (152h34m25.775968831s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.239576622 (152h34m25.791047084s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.283074694 (152h34m25.834545145s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.294108625 (152h34m25.845579086s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:47.996973416 (152h34m46.548443877s) [noise=true , address=145.79.227.21:4473 , piece_id=P7FNCN2PW445CVOUC6JBBUQYEC6ESHALJR2I5CCJ7ADB2R33OJJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:52.282269494 (152h34m50.833739955s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:54.127175860 (152h34m52.678646311s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection Nov 15 16:38:54.127196390 (152h34m52.678666851s) [noise=true , address=145.79.227.28:4726 , piece_id=DSSAU37SBLAVNIU3A46JPOGCNBTBT5B4AG767N5DA4OFZZZA3Y2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:38:54.127411158 (152h34m52.678881629s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection Nov 15 16:38:54.127524273 (152h34m52.678994725s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection Nov 15 16:38:54.127571524 (152h34m52.679041985s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:38:54.127613895 (152h34m52.679084356s) [noise=true , address=145.79.227.15:4874 , piece_id=JFR52BOF4X3JKDG4W2PWPALSICPMXUFW4ECMMJP3ZJ2EOZLXWFPQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection Nov 15 16:38:54.127679660 (152h34m52.679150111s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection Nov 15 16:38:54.127718884 (152h34m52.679189346s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection Nov 15 16:38:54.128019055 (152h34m52.679489516s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection Nov 15 16:38:54.128048231 (152h34m52.679518692s) [seg_pos={0 0}, upload=134380]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 16:38:54.375644018 (152h34m52.927114469s) [upload=134380]: Done uploading segments... Nov 15 16:38:54.375693672 (152h34m52.927164133s) [upload=134380]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 7.194144079s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................X...X......XX........] duration: 6.127549371s trace: https://jaeger.dev.storj.io/trace/6f5c529e7f01e189 FAIL (2025-11-15 16:44:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 07s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079179359 (152h34m23.63064982s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079536990 (152h34m23.631007451s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079909870 (152h34m23.631380351s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080308579 (152h34m23.631779041s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080706457 (152h34m23.632176908s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081011267 (152h34m23.632481728s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081354641 (152h34m23.632825102s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081749092 (152h34m23.633219543s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.082106943 (152h34m23.633577394s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084095681 (152h34m23.635566142s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084488068 (152h34m23.635958529s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084874634 (152h34m23.636345095s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:26.855580766 (152h34m25.407051217s) [upload=134380]: Next returned nil segment Nov 15 16:38:26.855606955 (152h34m25.407077416s) [upload=134380]: Waiting for error group managing segments... Nov 15 16:38:27.202416630 (152h34m25.753887081s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.224498370 (152h34m25.775968831s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.239576622 (152h34m25.791047084s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.283074694 (152h34m25.834545145s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.294108625 (152h34m25.845579086s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:47.996973416 (152h34m46.548443877s) [noise=true , address=145.79.227.21:4473 , piece_id=P7FNCN2PW445CVOUC6JBBUQYEC6ESHALJR2I5CCJ7ADB2R33OJJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:52.282269494 (152h34m50.833739955s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:54.127175860 (152h34m52.678646311s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection Nov 15 16:38:54.127196390 (152h34m52.678666851s) [noise=true , address=145.79.227.28:4726 , piece_id=DSSAU37SBLAVNIU3A46JPOGCNBTBT5B4AG767N5DA4OFZZZA3Y2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:38:54.127411158 (152h34m52.678881629s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection Nov 15 16:38:54.127524273 (152h34m52.678994725s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection Nov 15 16:38:54.127571524 (152h34m52.679041985s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:38:54.127613895 (152h34m52.679084356s) [noise=true , address=145.79.227.15:4874 , piece_id=JFR52BOF4X3JKDG4W2PWPALSICPMXUFW4ECMMJP3ZJ2EOZLXWFPQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection Nov 15 16:38:54.127679660 (152h34m52.679150111s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection Nov 15 16:38:54.127718884 (152h34m52.679189346s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection Nov 15 16:38:54.128019055 (152h34m52.679489516s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection Nov 15 16:38:54.128048231 (152h34m52.679518692s) [seg_pos={0 0}, upload=134380]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 16:38:54.375644018 (152h34m52.927114469s) [upload=134380]: Done uploading segments... Nov 15 16:38:54.375693672 (152h34m52.927164133s) [upload=134380]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.127578898s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................X...X......XX.......] duration: 21.749296216s trace: https://jaeger.dev.storj.io/trace/6f5c529e7f01e189 FAIL (2025-11-15 16:44:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 07s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079179359 (152h34m23.63064982s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079536990 (152h34m23.631007451s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079909870 (152h34m23.631380351s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080308579 (152h34m23.631779041s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080706457 (152h34m23.632176908s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081011267 (152h34m23.632481728s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081354641 (152h34m23.632825102s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081749092 (152h34m23.633219543s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.082106943 (152h34m23.633577394s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084095681 (152h34m23.635566142s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084488068 (152h34m23.635958529s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084874634 (152h34m23.636345095s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:26.855580766 (152h34m25.407051217s) [upload=134380]: Next returned nil segment Nov 15 16:38:26.855606955 (152h34m25.407077416s) [upload=134380]: Waiting for error group managing segments... Nov 15 16:38:27.202416630 (152h34m25.753887081s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.224498370 (152h34m25.775968831s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.239576622 (152h34m25.791047084s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.283074694 (152h34m25.834545145s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.294108625 (152h34m25.845579086s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:47.996973416 (152h34m46.548443877s) [noise=true , address=145.79.227.21:4473 , piece_id=P7FNCN2PW445CVOUC6JBBUQYEC6ESHALJR2I5CCJ7ADB2R33OJJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:52.282269494 (152h34m50.833739955s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:54.127175860 (152h34m52.678646311s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection Nov 15 16:38:54.127196390 (152h34m52.678666851s) [noise=true , address=145.79.227.28:4726 , piece_id=DSSAU37SBLAVNIU3A46JPOGCNBTBT5B4AG767N5DA4OFZZZA3Y2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:38:54.127411158 (152h34m52.678881629s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection Nov 15 16:38:54.127524273 (152h34m52.678994725s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection Nov 15 16:38:54.127571524 (152h34m52.679041985s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:38:54.127613895 (152h34m52.679084356s) [noise=true , address=145.79.227.15:4874 , piece_id=JFR52BOF4X3JKDG4W2PWPALSICPMXUFW4ECMMJP3ZJ2EOZLXWFPQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection Nov 15 16:38:54.127679660 (152h34m52.679150111s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection Nov 15 16:38:54.127718884 (152h34m52.679189346s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection Nov 15 16:38:54.128019055 (152h34m52.679489516s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection Nov 15 16:38:54.128048231 (152h34m52.679518692s) [seg_pos={0 0}, upload=134380]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 16:38:54.375644018 (152h34m52.927114469s) [upload=134380]: Done uploading segments... Nov 15 16:38:54.375693672 (152h34m52.927164133s) [upload=134380]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 21.749327144s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........X........X........................................] duration: 3.426848689s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:44:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.426887141s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........X........X.......................................] duration: 3.515564399s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:43:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.515591571s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............X........X......................................] duration: 3.48436142s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:43:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.48441368s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................................X...X......XX......] duration: 6.269687588s trace: https://jaeger.dev.storj.io/trace/6f5c529e7f01e189 FAIL (2025-11-15 16:42:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 07s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079179359 (152h34m23.63064982s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079536990 (152h34m23.631007451s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079909870 (152h34m23.631380351s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080308579 (152h34m23.631779041s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080706457 (152h34m23.632176908s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081011267 (152h34m23.632481728s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081354641 (152h34m23.632825102s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081749092 (152h34m23.633219543s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.082106943 (152h34m23.633577394s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084095681 (152h34m23.635566142s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084488068 (152h34m23.635958529s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084874634 (152h34m23.636345095s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:26.855580766 (152h34m25.407051217s) [upload=134380]: Next returned nil segment Nov 15 16:38:26.855606955 (152h34m25.407077416s) [upload=134380]: Waiting for error group managing segments... Nov 15 16:38:27.202416630 (152h34m25.753887081s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.224498370 (152h34m25.775968831s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.239576622 (152h34m25.791047084s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.283074694 (152h34m25.834545145s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.294108625 (152h34m25.845579086s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:47.996973416 (152h34m46.548443877s) [noise=true , address=145.79.227.21:4473 , piece_id=P7FNCN2PW445CVOUC6JBBUQYEC6ESHALJR2I5CCJ7ADB2R33OJJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:52.282269494 (152h34m50.833739955s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:54.127175860 (152h34m52.678646311s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection Nov 15 16:38:54.127196390 (152h34m52.678666851s) [noise=true , address=145.79.227.28:4726 , piece_id=DSSAU37SBLAVNIU3A46JPOGCNBTBT5B4AG767N5DA4OFZZZA3Y2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:38:54.127411158 (152h34m52.678881629s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection Nov 15 16:38:54.127524273 (152h34m52.678994725s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection Nov 15 16:38:54.127571524 (152h34m52.679041985s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:38:54.127613895 (152h34m52.679084356s) [noise=true , address=145.79.227.15:4874 , piece_id=JFR52BOF4X3JKDG4W2PWPALSICPMXUFW4ECMMJP3ZJ2EOZLXWFPQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection Nov 15 16:38:54.127679660 (152h34m52.679150111s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection Nov 15 16:38:54.127718884 (152h34m52.679189346s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection Nov 15 16:38:54.128019055 (152h34m52.679489516s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection Nov 15 16:38:54.128048231 (152h34m52.679518692s) [seg_pos={0 0}, upload=134380]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 16:38:54.375644018 (152h34m52.927114469s) [upload=134380]: Done uploading segments... Nov 15 16:38:54.375693672 (152h34m52.927164133s) [upload=134380]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.26971518s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............X........X.....................................] duration: 3.468322358s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:42:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.46838702s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................X...X......XX.....] duration: 6.20684282s trace: https://jaeger.dev.storj.io/trace/6f5c529e7f01e189 FAIL (2025-11-15 16:41:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 07s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079179359 (152h34m23.63064982s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079536990 (152h34m23.631007451s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079909870 (152h34m23.631380351s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080308579 (152h34m23.631779041s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080706457 (152h34m23.632176908s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081011267 (152h34m23.632481728s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081354641 (152h34m23.632825102s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081749092 (152h34m23.633219543s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.082106943 (152h34m23.633577394s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084095681 (152h34m23.635566142s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084488068 (152h34m23.635958529s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084874634 (152h34m23.636345095s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:26.855580766 (152h34m25.407051217s) [upload=134380]: Next returned nil segment Nov 15 16:38:26.855606955 (152h34m25.407077416s) [upload=134380]: Waiting for error group managing segments... Nov 15 16:38:27.202416630 (152h34m25.753887081s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.224498370 (152h34m25.775968831s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.239576622 (152h34m25.791047084s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.283074694 (152h34m25.834545145s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.294108625 (152h34m25.845579086s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:47.996973416 (152h34m46.548443877s) [noise=true , address=145.79.227.21:4473 , piece_id=P7FNCN2PW445CVOUC6JBBUQYEC6ESHALJR2I5CCJ7ADB2R33OJJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:52.282269494 (152h34m50.833739955s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:54.127175860 (152h34m52.678646311s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection Nov 15 16:38:54.127196390 (152h34m52.678666851s) [noise=true , address=145.79.227.28:4726 , piece_id=DSSAU37SBLAVNIU3A46JPOGCNBTBT5B4AG767N5DA4OFZZZA3Y2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:38:54.127411158 (152h34m52.678881629s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection Nov 15 16:38:54.127524273 (152h34m52.678994725s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection Nov 15 16:38:54.127571524 (152h34m52.679041985s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:38:54.127613895 (152h34m52.679084356s) [noise=true , address=145.79.227.15:4874 , piece_id=JFR52BOF4X3JKDG4W2PWPALSICPMXUFW4ECMMJP3ZJ2EOZLXWFPQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection Nov 15 16:38:54.127679660 (152h34m52.679150111s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection Nov 15 16:38:54.127718884 (152h34m52.679189346s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection Nov 15 16:38:54.128019055 (152h34m52.679489516s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection Nov 15 16:38:54.128048231 (152h34m52.679518692s) [seg_pos={0 0}, upload=134380]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 16:38:54.375644018 (152h34m52.927114469s) [upload=134380]: Done uploading segments... Nov 15 16:38:54.375693672 (152h34m52.927164133s) [upload=134380]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.206871003s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............X........X....................................] duration: 3.516301932s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:41:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.516329907s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................................X...X......XX....] duration: 6.118134644s trace: https://jaeger.dev.storj.io/trace/6f5c529e7f01e189 FAIL (2025-11-15 16:41:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 07s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079179359 (152h34m23.63064982s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079536990 (152h34m23.631007451s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079909870 (152h34m23.631380351s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080308579 (152h34m23.631779041s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080706457 (152h34m23.632176908s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081011267 (152h34m23.632481728s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081354641 (152h34m23.632825102s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081749092 (152h34m23.633219543s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.082106943 (152h34m23.633577394s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084095681 (152h34m23.635566142s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084488068 (152h34m23.635958529s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084874634 (152h34m23.636345095s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:26.855580766 (152h34m25.407051217s) [upload=134380]: Next returned nil segment Nov 15 16:38:26.855606955 (152h34m25.407077416s) [upload=134380]: Waiting for error group managing segments... Nov 15 16:38:27.202416630 (152h34m25.753887081s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.224498370 (152h34m25.775968831s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.239576622 (152h34m25.791047084s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.283074694 (152h34m25.834545145s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.294108625 (152h34m25.845579086s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:47.996973416 (152h34m46.548443877s) [noise=true , address=145.79.227.21:4473 , piece_id=P7FNCN2PW445CVOUC6JBBUQYEC6ESHALJR2I5CCJ7ADB2R33OJJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:52.282269494 (152h34m50.833739955s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:54.127175860 (152h34m52.678646311s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection Nov 15 16:38:54.127196390 (152h34m52.678666851s) [noise=true , address=145.79.227.28:4726 , piece_id=DSSAU37SBLAVNIU3A46JPOGCNBTBT5B4AG767N5DA4OFZZZA3Y2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:38:54.127411158 (152h34m52.678881629s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection Nov 15 16:38:54.127524273 (152h34m52.678994725s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection Nov 15 16:38:54.127571524 (152h34m52.679041985s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:38:54.127613895 (152h34m52.679084356s) [noise=true , address=145.79.227.15:4874 , piece_id=JFR52BOF4X3JKDG4W2PWPALSICPMXUFW4ECMMJP3ZJ2EOZLXWFPQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection Nov 15 16:38:54.127679660 (152h34m52.679150111s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection Nov 15 16:38:54.127718884 (152h34m52.679189346s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection Nov 15 16:38:54.128019055 (152h34m52.679489516s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection Nov 15 16:38:54.128048231 (152h34m52.679518692s) [seg_pos={0 0}, upload=134380]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 16:38:54.375644018 (152h34m52.927114469s) [upload=134380]: Done uploading segments... Nov 15 16:38:54.375693672 (152h34m52.927164133s) [upload=134380]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.118151105s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............X........X...................................] duration: 3.48144592s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:40:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.481476788s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................X...X......XX...] duration: 6.3650625s trace: https://jaeger.dev.storj.io/trace/6f5c529e7f01e189 FAIL (2025-11-15 16:40:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 07s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079179359 (152h34m23.63064982s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079536990 (152h34m23.631007451s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079909870 (152h34m23.631380351s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080308579 (152h34m23.631779041s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080706457 (152h34m23.632176908s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081011267 (152h34m23.632481728s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081354641 (152h34m23.632825102s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081749092 (152h34m23.633219543s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.082106943 (152h34m23.633577394s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084095681 (152h34m23.635566142s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084488068 (152h34m23.635958529s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084874634 (152h34m23.636345095s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:26.855580766 (152h34m25.407051217s) [upload=134380]: Next returned nil segment Nov 15 16:38:26.855606955 (152h34m25.407077416s) [upload=134380]: Waiting for error group managing segments... Nov 15 16:38:27.202416630 (152h34m25.753887081s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.224498370 (152h34m25.775968831s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.239576622 (152h34m25.791047084s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.283074694 (152h34m25.834545145s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.294108625 (152h34m25.845579086s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:47.996973416 (152h34m46.548443877s) [noise=true , address=145.79.227.21:4473 , piece_id=P7FNCN2PW445CVOUC6JBBUQYEC6ESHALJR2I5CCJ7ADB2R33OJJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:52.282269494 (152h34m50.833739955s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:54.127175860 (152h34m52.678646311s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection Nov 15 16:38:54.127196390 (152h34m52.678666851s) [noise=true , address=145.79.227.28:4726 , piece_id=DSSAU37SBLAVNIU3A46JPOGCNBTBT5B4AG767N5DA4OFZZZA3Y2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:38:54.127411158 (152h34m52.678881629s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection Nov 15 16:38:54.127524273 (152h34m52.678994725s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection Nov 15 16:38:54.127571524 (152h34m52.679041985s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:38:54.127613895 (152h34m52.679084356s) [noise=true , address=145.79.227.15:4874 , piece_id=JFR52BOF4X3JKDG4W2PWPALSICPMXUFW4ECMMJP3ZJ2EOZLXWFPQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection Nov 15 16:38:54.127679660 (152h34m52.679150111s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection Nov 15 16:38:54.127718884 (152h34m52.679189346s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection Nov 15 16:38:54.128019055 (152h34m52.679489516s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection Nov 15 16:38:54.128048231 (152h34m52.679518692s) [seg_pos={0 0}, upload=134380]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 16:38:54.375644018 (152h34m52.927114469s) [upload=134380]: Done uploading segments... Nov 15 16:38:54.375693672 (152h34m52.927164133s) [upload=134380]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.365091505s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................................X...X......XX..] duration: 6.23463097s trace: https://jaeger.dev.storj.io/trace/6f5c529e7f01e189 FAIL (2025-11-15 16:40:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 07s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079179359 (152h34m23.63064982s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079536990 (152h34m23.631007451s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079909870 (152h34m23.631380351s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080308579 (152h34m23.631779041s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080706457 (152h34m23.632176908s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081011267 (152h34m23.632481728s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081354641 (152h34m23.632825102s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081749092 (152h34m23.633219543s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.082106943 (152h34m23.633577394s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084095681 (152h34m23.635566142s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084488068 (152h34m23.635958529s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084874634 (152h34m23.636345095s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:26.855580766 (152h34m25.407051217s) [upload=134380]: Next returned nil segment Nov 15 16:38:26.855606955 (152h34m25.407077416s) [upload=134380]: Waiting for error group managing segments... Nov 15 16:38:27.202416630 (152h34m25.753887081s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.224498370 (152h34m25.775968831s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.239576622 (152h34m25.791047084s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.283074694 (152h34m25.834545145s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.294108625 (152h34m25.845579086s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:47.996973416 (152h34m46.548443877s) [noise=true , address=145.79.227.21:4473 , piece_id=P7FNCN2PW445CVOUC6JBBUQYEC6ESHALJR2I5CCJ7ADB2R33OJJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:52.282269494 (152h34m50.833739955s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:54.127175860 (152h34m52.678646311s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection Nov 15 16:38:54.127196390 (152h34m52.678666851s) [noise=true , address=145.79.227.28:4726 , piece_id=DSSAU37SBLAVNIU3A46JPOGCNBTBT5B4AG767N5DA4OFZZZA3Y2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:38:54.127411158 (152h34m52.678881629s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection Nov 15 16:38:54.127524273 (152h34m52.678994725s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection Nov 15 16:38:54.127571524 (152h34m52.679041985s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:38:54.127613895 (152h34m52.679084356s) [noise=true , address=145.79.227.15:4874 , piece_id=JFR52BOF4X3JKDG4W2PWPALSICPMXUFW4ECMMJP3ZJ2EOZLXWFPQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection Nov 15 16:38:54.127679660 (152h34m52.679150111s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection Nov 15 16:38:54.127718884 (152h34m52.679189346s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection Nov 15 16:38:54.128019055 (152h34m52.679489516s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection Nov 15 16:38:54.128048231 (152h34m52.679518692s) [seg_pos={0 0}, upload=134380]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 16:38:54.375644018 (152h34m52.927114469s) [upload=134380]: Done uploading segments... Nov 15 16:38:54.375693672 (152h34m52.927164133s) [upload=134380]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.234685193s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................X........X..................................] duration: 3.369926687s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:39:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.369956703s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................X........X.................................] duration: 3.49978827s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:39:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.499818299s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................X...X......XX.] duration: 6.964799301s trace: https://jaeger.dev.storj.io/trace/6f5c529e7f01e189 FAIL (2025-11-15 16:39:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 07s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079179359 (152h34m23.63064982s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079536990 (152h34m23.631007451s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079909870 (152h34m23.631380351s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080308579 (152h34m23.631779041s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080706457 (152h34m23.632176908s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081011267 (152h34m23.632481728s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081354641 (152h34m23.632825102s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081749092 (152h34m23.633219543s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.082106943 (152h34m23.633577394s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084095681 (152h34m23.635566142s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084488068 (152h34m23.635958529s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084874634 (152h34m23.636345095s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:26.855580766 (152h34m25.407051217s) [upload=134380]: Next returned nil segment Nov 15 16:38:26.855606955 (152h34m25.407077416s) [upload=134380]: Waiting for error group managing segments... Nov 15 16:38:27.202416630 (152h34m25.753887081s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.224498370 (152h34m25.775968831s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.239576622 (152h34m25.791047084s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.283074694 (152h34m25.834545145s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.294108625 (152h34m25.845579086s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:47.996973416 (152h34m46.548443877s) [noise=true , address=145.79.227.21:4473 , piece_id=P7FNCN2PW445CVOUC6JBBUQYEC6ESHALJR2I5CCJ7ADB2R33OJJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:52.282269494 (152h34m50.833739955s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:54.127175860 (152h34m52.678646311s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection Nov 15 16:38:54.127196390 (152h34m52.678666851s) [noise=true , address=145.79.227.28:4726 , piece_id=DSSAU37SBLAVNIU3A46JPOGCNBTBT5B4AG767N5DA4OFZZZA3Y2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:38:54.127411158 (152h34m52.678881629s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection Nov 15 16:38:54.127524273 (152h34m52.678994725s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection Nov 15 16:38:54.127571524 (152h34m52.679041985s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:38:54.127613895 (152h34m52.679084356s) [noise=true , address=145.79.227.15:4874 , piece_id=JFR52BOF4X3JKDG4W2PWPALSICPMXUFW4ECMMJP3ZJ2EOZLXWFPQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection Nov 15 16:38:54.127679660 (152h34m52.679150111s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection Nov 15 16:38:54.127718884 (152h34m52.679189346s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection Nov 15 16:38:54.128019055 (152h34m52.679489516s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection Nov 15 16:38:54.128048231 (152h34m52.679518692s) [seg_pos={0 0}, upload=134380]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 16:38:54.375644018 (152h34m52.927114469s) [upload=134380]: Done uploading segments... Nov 15 16:38:54.375693672 (152h34m52.927164133s) [upload=134380]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 6.964820772s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................................X...X......XX] duration: 30.250169427s trace: https://jaeger.dev.storj.io/trace/6f5c529e7f01e189 FAIL (2025-11-15 16:38:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 07s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079179359 (152h34m23.63064982s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079536990 (152h34m23.631007451s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.079909870 (152h34m23.631380351s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080308579 (152h34m23.631779041s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.080706457 (152h34m23.632176908s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081011267 (152h34m23.632481728s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081354641 (152h34m23.632825102s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.081749092 (152h34m23.633219543s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.082106943 (152h34m23.633577394s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084095681 (152h34m23.635566142s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084488068 (152h34m23.635958529s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:25.084874634 (152h34m23.636345095s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Uploading piece... Nov 15 16:38:26.855580766 (152h34m25.407051217s) [upload=134380]: Next returned nil segment Nov 15 16:38:26.855606955 (152h34m25.407077416s) [upload=134380]: Waiting for error group managing segments... Nov 15 16:38:27.202416630 (152h34m25.753887081s) [noise=true , address=77.90.31.12:4590 , piece_id=3AUADPKYMPGYCAC7VLSP7ZWW25FSSPZK452RTMSRVR6DYTIJYWQA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.224498370 (152h34m25.775968831s) [noise=true , address=45.91.116.4:4939 , piece_id=DQOLESUDDY4NE4ONPAC53XT4OVF7RX6QHD65ONIOIB6ZBYIIQ22A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.239576622 (152h34m25.791047084s) [noise=true , address=45.91.116.13:4153 , piece_id=UJMDATEIFVV5JQYAKKS6EQRJZ4K5W2CXILEZY36UAATWZM67JEJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.283074694 (152h34m25.834545145s) [noise=true , address=77.90.31.10:4519 , piece_id=MYSZTG5PBL6XGBZSB54TI6BLOAE44GVI2NENGNVCRRPIICSH6W2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:27.294108625 (152h34m25.845579086s) [noise=true , address=45.91.116.5:4298 , piece_id=HFNEFVECOWRB2D5FKP3IWQNX3HTTC5YLORY6SEO5QWXFKRLMQ4XQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:47.996973416 (152h34m46.548443877s) [noise=true , address=145.79.227.21:4473 , piece_id=P7FNCN2PW445CVOUC6JBBUQYEC6ESHALJR2I5CCJ7ADB2R33OJJQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:52.282269494 (152h34m50.833739955s) [noise=true , address=145.79.227.25:4618 , piece_id=A75MTVBX35CNJPI7TTBWJOEEVBRSAORSMD6Z7SR25ANDXZ42CZFQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: <nil> Nov 15 16:38:54.127175860 (152h34m52.678646311s) [noise=true , address=145.79.227.10:4403 , piece_id=EGZFZJ7VWZEBQOHZHHB3SIPMMIRKBWOSL5ZYAWSUYA76XOYCSLRQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57412->145.79.227.10:4403: use of closed network connection Nov 15 16:38:54.127196390 (152h34m52.678666851s) [noise=true , address=145.79.227.28:4726 , piece_id=DSSAU37SBLAVNIU3A46JPOGCNBTBT5B4AG767N5DA4OFZZZA3Y2A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:38:54.127411158 (152h34m52.678881629s) [noise=true , address=145.79.227.22:4511 , piece_id=WXDJL4DC5OFZYZRGI23UZE6VA5QQB6YQOBC7BZGDLJMI4H65WB4A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46734->145.79.227.22:4511: use of closed network connection Nov 15 16:38:54.127524273 (152h34m52.678994725s) [noise=true , address=145.79.227.17:4970 , piece_id=PBGKUOLODXOVLT3OO5XZI4PKVUQNDAI2UAKA62WR4UGDSQTDIFRA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44942->145.79.227.17:4970: use of closed network connection Nov 15 16:38:54.127571524 (152h34m52.679041985s) [noise=true , address=145.79.227.7:4295 , piece_id=ETMFTYJGL4444HNQYIFU2VCICQRAEZUQ5JGTWJJUX6SKTMOAL7NA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:38:54.127613895 (152h34m52.679084356s) [noise=true , address=145.79.227.15:4874 , piece_id=JFR52BOF4X3JKDG4W2PWPALSICPMXUFW4ECMMJP3ZJ2EOZLXWFPQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35762->145.79.227.15:4874: use of closed network connection Nov 15 16:38:54.127679660 (152h34m52.679150111s) [noise=true , address=145.79.227.8:4330 , piece_id=4MIT2ENWE6IKIO5H5JUNNWGEBHBWQATZWW7GFVEUYPXXOFZ7CJWA, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49830->145.79.227.8:4330: use of closed network connection Nov 15 16:38:54.127718884 (152h34m52.679189346s) [noise=true , address=145.79.227.11:4732 , piece_id=TFA4PWYDTDKMDSO4UAJNIEPESNE3PWFPOMLQNBMLAKCKJXBMV43A, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35014->145.79.227.11:4732: use of closed network connection Nov 15 16:38:54.128019055 (152h34m52.679489516s) [noise=true , address=145.79.227.16:4912 , piece_id=T6VCYDFVO6BLESBYG4SS5QPSHSCZYSCICAGYN6UJXTGQLEK7TSDQ, seg_pos={0 0}, upload=134380]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43914->145.79.227.16:4912: use of closed network connection Nov 15 16:38:54.128048231 (152h34m52.679518692s) [seg_pos={0 0}, upload=134380]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Nov 15 16:38:54.375644018 (152h34m52.927114469s) [upload=134380]: Done uploading segments... Nov 15 16:38:54.375693672 (152h34m52.927164133s) [upload=134380]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.25018113s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................X...X......X] duration: 30.244218284s trace: https://jaeger.dev.storj.io/trace/17411d80b0950e94 FAIL (2025-11-15 16:38:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 15 16:37:54.350168274 (152h33m52.901638735s) [upload=134371]: Starting upload Nov 15 16:37:54.350240030 (152h33m52.901710491s) [upload=134371]: Done starting upload Nov 15 16:37:54.350461212 (152h33m52.901931673s) [upload=134371]: Uploading segments... Nov 15 16:37:54.350503552 (152h33m52.901974014s) [upload=134371]: Got next segment. Inline: false Nov 15 16:37:54.350532738 (152h33m52.902003199s) [upload=134371]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 16:37:54.350537998 (152h33m52.902008459s) [upload=134371]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 16:37:54.603044585 (152h33m53.154515047s) [seg_pos={0 0}, upload=134371]: Begin upload segment... Nov 15 16:37:54.603106403 (152h33m53.154576864s) [seg_pos={0 0}, upload=134371]: Done begin upload segment. Nov 15 16:37:54.603140839 (152h33m53.1546113s) [noise=true , address=145.79.227.22:4511 , piece_id=XOPXILG6KDJJPLUURKXDIQMMOE255PHCF4MHGQV575FU4J7U476Q, seg_pos={0 0}, upload=134371]: Uploading piece... Nov 15 16:37:54.603584012 (152h33m53.155054503s) [noise=true , address=145.79.227.32:4836 , piece_id=GSDDKIYL4HILD54FIYJGE7UV636DFO6LBMV6K756JZ6VZOMZ4GUA, seg_pos={0 0}, upload=134371]: Uploading piece... Nov 15 16:37:54.603957474 (152h33m53.155427935s) [noise=true , address=45.91.116.5:4298 , piece_id=77YJGRODM46UW3R3A5J4DZGXZQJNRYHZWLJQT2T3BDMDLQQ7XYKA, seg_pos={0 0}, upload=134371]: Uploading piece... Nov 15 16:37:54.604721929 (152h33m53.15619239s) [noise=true , address=145.79.227.15:4876 , piece_id=ZDLALTPAWODUOSXL3URYIMZTD633HGV4EDT5W33LBKY3CIT7QOYA, seg_pos={0 0}, upload=134371]: Uploading piece... Nov 15 16:37:54.605244003 (152h33m53.156714464s) [noise=true , address=77.90.31.16:4734 , piece_id=65Y7ZFHGBCTIBAYFB6PEJ343L4BPOMQMDNGCHT42SXGNOWI4ABDQ, seg_pos={0 0}, upload=134371]: Uploading piece... Nov 15 16:37:54.605426993 (152h33m53.156897454s) [noise=true , address=145.79.227.20:4439 , piece_id=5JM37KZP345774DLHE5YUJVTNLLUVDEXOZUNEWLRMR374Y2OU64A, seg_pos={0 0}, upload=134371]: Uploading piece... Nov 15 16:37:54.605718898 (152h33m53.157189349s) [noise=true , address=77.90.31.13:4627 , piece_id=P7ARYZ6KIQT4OA3S5GZT4ZLSKA6CLBFIMX5XFEXNWYIMF253PHWA, seg_pos={0 0}, upload=134371]: Uploading piece... Nov 15 16:37:54.606002648 (152h33m53.157473099s) [noise=true , address=145.79.227.21:4474 , piece_id=G4VKNROD2IIXOVGKMJZKZHL6OGSH2OR5WPHHXLZOARAKT75SGOBQ, seg_pos={0 0}, upload=134371]: Uploading piece... Nov 15 16:37:54.606209021 (152h33m53.157679482s) [noise=true , address=145.79.227.8:4330 , piece_id=PFV54MUCN5RM4JXYDC5M3WA2K5ILYRDI3OYK4W6DD3WPAD3GH5RQ, seg_pos={0 0}, upload=134371]: Uploading piece... Nov 15 16:37:54.606514383 (152h33m53.157984844s) [noise=true , address=145.79.227.18:4005 , piece_id=IT5B7EGD3HBJNL32ODBETDCMX43NMC4PHGVOWMH4AJS7DCCG3HMA, seg_pos={0 0}, upload=134371]: Uploading piece... Nov 15 16:37:54.606631566 (152h33m53.158102017s) [noise=true , address=145.79.227.13:4804 , piece_id=2YTW5U67HHNIL4G2AIE3ZKBD7LZK2KZ5BVMYNGKXZDOVQTKIOYNA, seg_pos={0 0}, upload=134371]: Uploading piece... Nov 15 16:37:54.607033892 (152h33m53.158504353s) [noise=true , address=145.79.227.11:4730 , piece_id=2YMOZKSEVOXITY3LIOQL66DOYVE336HD332QGQFWD6ZBYBN2UGNQ, seg_pos={0 0}, upload=134371]: Uploading piece... Nov 15 16:37:54.607199678 (152h33m53.158670129s) [noise=true , address=77.90.31.12:4589 , piece_id=FSLTEAHIUDP3NVH7NIZGLLV5DQOUAHTXP23B2QKI654AZIXLUCOA, seg_pos={0 0}, upload=134371]: Uploading piece... Nov 15 16:37:54.607596794 (152h33m53.159067245s) [noise=true , address=145.79.227.17:4971 , piece_id=IWWG2M6AB56WUHTBKAPBMQ6F3EEV6CIRHT6GV4NUAAOUBZUHXJAA, seg_pos={0 0}, upload=134371]: Uploading piece... Nov 15 16:37:54.607915541 (152h33m53.159386032s) [noise=true , address=77.90.31.10:4519 , piece_id=VILUTSRDWXZIZZKBO4E62TT7B74LEXBZPN6FFXYZOLVEZTBQKTEQ, seg_pos={0 0}, upload=134371]: Uploading piece... Nov 15 16:37:54.608214330 (152h33m53.159684791s) [noise=true , address=145.79.227.14:4837 , piece_id=FIHZUVCR5QMTPUP7BAVWAIHMDTH4NCAZSI7KXOJVGBPMYZRNG6PQ, seg_pos={0 0}, upload=134371]: Uploading piece... Nov 15 16:37:56.246051182 (152h33m54.797521643s) [upload=134371]: Next returned nil segment Nov 15 16:37:56.246065550 (152h33m54.797536011s) [upload=134371]: Waiting for error group managing segments... Nov 15 16:37:56.657163195 (152h33m55.208633656s) [noise=true , address=145.79.227.13:4804 , piece_id=2YTW5U67HHNIL4G2AIE3ZKBD7LZK2KZ5BVMYNGKXZDOVQTKIOYNA, seg_pos={0 0}, upload=134371]: Done uploading piece. err: <nil> Nov 15 16:37:56.670842394 (152h33m55.222312855s) [noise=true , address=145.79.227.20:4439 , piece_id=5JM37KZP345774DLHE5YUJVTNLLUVDEXOZUNEWLRMR374Y2OU64A, seg_pos={0 0}, upload=134371]: Done uploading piece. err: <nil> Nov 15 16:37:56.678584707 (152h33m55.230055168s) [noise=true , address=145.79.227.21:4474 , piece_id=G4VKNROD2IIXOVGKMJZKZHL6OGSH2OR5WPHHXLZOARAKT75SGOBQ, seg_pos={0 0}, upload=134371]: Done uploading piece. err: <nil> Nov 15 16:37:56.679625369 (152h33m55.23109583s) [noise=true , address=145.79.227.18:4005 , piece_id=IT5B7EGD3HBJNL32ODBETDCMX43NMC4PHGVOWMH4AJS7DCCG3HMA, seg_pos={0 0}, upload=134371]: Done uploading piece. err: <nil> Nov 15 16:37:56.709667147 (152h33m55.261137608s) [noise=true , address=45.91.116.5:4298 , piece_id=77YJGRODM46UW3R3A5J4DZGXZQJNRYHZWLJQT2T3BDMDLQQ7XYKA, seg_pos={0 0}, upload=134371]: Done uploading piece. err: <nil> Nov 15 16:37:56.713159117 (152h33m55.264629568s) [noise=true , address=77.90.31.13:4627 , piece_id=P7ARYZ6KIQT4OA3S5GZT4ZLSKA6CLBFIMX5XFEXNWYIMF253PHWA, seg_pos={0 0}, upload=134371]: Done uploading piece. err: <nil> Nov 15 16:37:56.790394302 (152h33m55.341864763s) [noise=true , address=77.90.31.16:4734 , piece_id=65Y7ZFHGBCTIBAYFB6PEJ343L4BPOMQMDNGCHT42SXGNOWI4ABDQ, seg_pos={0 0}, upload=134371]: Done uploading piece. err: <nil> Nov 15 16:37:56.909371004 (152h33m55.460841476s) [noise=true , address=77.90.31.10:4519 , piece_id=VILUTSRDWXZIZZKBO4E62TT7B74LEXBZPN6FFXYZOLVEZTBQKTEQ, seg_pos={0 0}, upload=134371]: Done uploading piece. err: <nil> Nov 15 16:37:57.156250466 (152h33m55.707720918s) [noise=true , address=77.90.31.12:4589 , piece_id=FSLTEAHIUDP3NVH7NIZGLLV5DQOUAHTXP23B2QKI654AZIXLUCOA, seg_pos={0 0}, upload=134371]: Done uploading piece. err: <nil> Nov 15 16:37:58.132982084 (152h33m56.684452545s) [noise=true , address=145.79.227.17:4971 , piece_id=IWWG2M6AB56WUHTBKAPBMQ6F3EEV6CIRHT6GV4NUAAOUBZUHXJAA, seg_pos={0 0}, upload=134371]: Done uploading piece. err: <nil> Nov 15 16:38:01.843038155 (152h34m0.394508656s) [noise=true , address=145.79.227.14:4837 , piece_id=FIHZUVCR5QMTPUP7BAVWAIHMDTH4NCAZSI7KXOJVGBPMYZRNG6PQ, seg_pos={0 0}, upload=134371]: Done uploading piece. err: <nil> Nov 15 16:38:01.850522517 (152h34m0.401992978s) [noise=true , address=145.79.227.15:4876 , piece_id=ZDLALTPAWODUOSXL3URYIMZTD633HGV4EDT5W33LBKY3CIT7QOYA, seg_pos={0 0}, upload=134371]: Done uploading piece. err: <nil> Nov 15 16:38:23.654548876 (152h34m22.206019327s) [noise=true , address=145.79.227.8:4330 , piece_id=PFV54MUCN5RM4JXYDC5M3WA2K5ILYRDI3OYK4W6DD3WPAD3GH5RQ, seg_pos={0 0}, upload=134371]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:38:23.654687009 (152h34m22.20615747s) [noise=true , address=145.79.227.22:4511 , piece_id=XOPXILG6KDJJPLUURKXDIQMMOE255PHCF4MHGQV575FU4J7U476Q, seg_pos={0 0}, upload=134371]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:38:23.654790025 (152h34m22.206260486s) [noise=true , address=145.79.227.32:4836 , piece_id=GSDDKIYL4HILD54FIYJGE7UV636DFO6LBMV6K756JZ6VZOMZ4GUA, seg_pos={0 0}, upload=134371]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:38:23.654890436 (152h34m22.206360887s) [noise=true , address=145.79.227.11:4730 , piece_id=2YMOZKSEVOXITY3LIOQL66DOYVE336HD332QGQFWD6ZBYBN2UGNQ, seg_pos={0 0}, upload=134371]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:38:23.654915043 (152h34m22.206385504s) [seg_pos={0 0}, upload=134371]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 15 16:38:23.896589233 (152h34m22.448059704s) [upload=134371]: Done uploading segments... Nov 15 16:38:23.896632415 (152h34m22.448102866s) [upload=134371]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.244237852s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................X........X................................] duration: 3.585866091s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:38:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.585884486s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X...X......] duration: 6.065587344s trace: https://jaeger.dev.storj.io/trace/5a8562300700bcde FAIL (2025-11-15 16:37:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== e=true , address=145.79.227.8:4329 , piece_id=BZNWOFF4FXQAB4GDDSJLG6ANC4UYY4F743U7E63MCYWEXD6O6R5A, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.851927882 (152h29m22.403398343s) [noise=true , address=45.91.116.14:4190 , piece_id=7KMDRAD6OXDEOZNHHMEWP6IWIB5KBDU6PNA4LPILDN5ZG7QBB6BA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.854137329 (152h29m22.40560779s) [noise=true , address=77.90.31.18:4806 , piece_id=QCWZNPVJANVBAHKV55FW7QYNU27OKAFMP5XDN2TDT7XVLMJMNQQA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.856035504 (152h29m22.407505965s) [noise=true , address=145.79.227.13:4804 , piece_id=Y7HA6XLUDC2VP7JEKPJNXKHFZJ77U5USNT5RCVW4MS2QZACBOY5Q, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.859311512 (152h29m22.410781973s) [noise=true , address=45.91.116.10:4082 , piece_id=ETWTGCVFPECYXLWSK6U5Q5PAJTKNX5CK2SJJVWCG4KEACZ4IAZOA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.859442551 (152h29m22.410913013s) [noise=true , address=45.91.116.5:4298 , piece_id=GKNZT2GUEIHHPWJQIEH5BSUWO2XXOM6EQ5KZX2JTZSMCNB74FQ4Q, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.861349383 (152h29m22.412819834s) [noise=true , address=45.91.116.10:4083 , piece_id=N7STKXM3QZCERQN6W5VU6K5LMBESIFDANG6MY67XL57TBY5JRK4A, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.861826902 (152h29m22.413297363s) [noise=true , address=145.79.227.31:4799 , piece_id=2X2RCSEPSBRGIQR4TYXYND35QOPWUFWTJMXFXFIYCX3DYMJ442FQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.863192743 (152h29m22.414663204s) [noise=true , address=145.79.227.28:4727 , piece_id=CLFAJR2TX5F2R22PRBV26GTY3FZAM3C3O3AG47CU6N6YRDHODLGQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.864499722 (152h29m22.415970183s) [noise=true , address=145.79.227.11:4732 , piece_id=4W7WUHZE5VJXXYB65GF5QVAGNN4SHLGAD7OES2D4G2JGOYUU2YLQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.864794413 (152h29m22.416264864s) [noise=true , address=145.79.227.26:4653 , piece_id=G64IAIWXOOOHG3M3XMOHXHCXZ2LWSQ2XAB7ALFPAFWKOWEXKYAMQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.866380714 (152h29m22.417851175s) [noise=true , address=45.91.116.4:4940 , piece_id=E6OZVZK4KPEZPVISBIDIR56ATOXKHXWAFRO4KTJ3AN3F5S542ZWQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.867347054 (152h29m22.418817515s) [noise=true , address=145.79.227.15:4873 , piece_id=XDTTIU2YYZ4YX2CR7GFATGW5QV7B6NLO7CP3P7LHE4UFLAFEDLMA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:25.673146503 (152h29m24.224616954s) [upload=134305]: Next returned nil segment Nov 15 16:33:25.673183183 (152h29m24.224653644s) [upload=134305]: Waiting for error group managing segments... Nov 15 16:33:25.957140584 (152h29m24.508611035s) [noise=true , address=45.91.116.10:4083 , piece_id=N7STKXM3QZCERQN6W5VU6K5LMBESIFDANG6MY67XL57TBY5JRK4A, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.001103691 (152h29m24.552574162s) [noise=true , address=45.91.116.4:4940 , piece_id=E6OZVZK4KPEZPVISBIDIR56ATOXKHXWAFRO4KTJ3AN3F5S542ZWQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.003521624 (152h29m24.554992086s) [noise=true , address=45.91.116.5:4298 , piece_id=GKNZT2GUEIHHPWJQIEH5BSUWO2XXOM6EQ5KZX2JTZSMCNB74FQ4Q, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.004722812 (152h29m24.556193293s) [noise=true , address=77.90.31.18:4806 , piece_id=QCWZNPVJANVBAHKV55FW7QYNU27OKAFMP5XDN2TDT7XVLMJMNQQA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.027847948 (152h29m24.579318419s) [noise=true , address=77.90.31.11:4553 , piece_id=M3MGFKZJXBQ5BGYJVLP5HQJ7KGK3BBYULTEGPY4XJQDBAG3T5UIQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.038988902 (152h29m24.590459363s) [noise=true , address=45.91.116.10:4082 , piece_id=ETWTGCVFPECYXLWSK6U5Q5PAJTKNX5CK2SJJVWCG4KEACZ4IAZOA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.049585892 (152h29m24.601056354s) [noise=true , address=45.91.116.14:4190 , piece_id=7KMDRAD6OXDEOZNHHMEWP6IWIB5KBDU6PNA4LPILDN5ZG7QBB6BA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:45.222994074 (152h29m43.774464535s) [noise=true , address=145.79.227.15:4873 , piece_id=XDTTIU2YYZ4YX2CR7GFATGW5QV7B6NLO7CP3P7LHE4UFLAFEDLMA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:52.891596883 (152h29m51.443067334s) [noise=true , address=145.79.227.7:4295 , piece_id=ELJRP3KI6F7ZFOMIYGWSTSAELMSI5H6OS4WU5R5UGA7BC3P3BEGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection Nov 15 16:33:52.891716481 (152h29m51.443186952s) [noise=true , address=145.79.227.8:4329 , piece_id=BZNWOFF4FXQAB4GDDSJLG6ANC4UYY4F743U7E63MCYWEXD6O6R5A, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection Nov 15 16:33:52.891805550 (152h29m51.443276001s) [noise=true , address=145.79.227.11:4732 , piece_id=4W7WUHZE5VJXXYB65GF5QVAGNN4SHLGAD7OES2D4G2JGOYUU2YLQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection Nov 15 16:33:52.891822112 (152h29m51.443292583s) [noise=true , address=145.79.227.31:4799 , piece_id=2X2RCSEPSBRGIQR4TYXYND35QOPWUFWTJMXFXFIYCX3DYMJ442FQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection Nov 15 16:33:52.891924026 (152h29m51.443394477s) [noise=true , address=145.79.227.32:4835 , piece_id=ZFXRT7D575I3NCZEENPTV7T4GEF6LEIBBTY6A5MBQ455ZTRDHYGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection Nov 15 16:33:52.892373372 (152h29m51.443843833s) [noise=true , address=145.79.227.28:4727 , piece_id=CLFAJR2TX5F2R22PRBV26GTY3FZAM3C3O3AG47CU6N6YRDHODLGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection Nov 15 16:33:52.892554417 (152h29m51.444024868s) [noise=true , address=145.79.227.26:4653 , piece_id=G64IAIWXOOOHG3M3XMOHXHCXZ2LWSQ2XAB7ALFPAFWKOWEXKYAMQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection Nov 15 16:33:52.892663404 (152h29m51.444133865s) [noise=true , address=145.79.227.13:4804 , piece_id=Y7HA6XLUDC2VP7JEKPJNXKHFZJ77U5USNT5RCVW4MS2QZACBOY5Q, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection Nov 15 16:33:52.892704532 (152h29m51.444174993s) [seg_pos={0 0}, upload=134305]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 15 16:33:53.136366081 (152h29m51.687836532s) [upload=134305]: Done uploading segments... Nov 15 16:33:53.136405987 (152h29m51.687876448s) [upload=134305]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.065614317s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X...X.....] duration: 5.827275405s trace: https://jaeger.dev.storj.io/trace/5a8562300700bcde FAIL (2025-11-15 16:37:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== e=true , address=145.79.227.8:4329 , piece_id=BZNWOFF4FXQAB4GDDSJLG6ANC4UYY4F743U7E63MCYWEXD6O6R5A, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.851927882 (152h29m22.403398343s) [noise=true , address=45.91.116.14:4190 , piece_id=7KMDRAD6OXDEOZNHHMEWP6IWIB5KBDU6PNA4LPILDN5ZG7QBB6BA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.854137329 (152h29m22.40560779s) [noise=true , address=77.90.31.18:4806 , piece_id=QCWZNPVJANVBAHKV55FW7QYNU27OKAFMP5XDN2TDT7XVLMJMNQQA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.856035504 (152h29m22.407505965s) [noise=true , address=145.79.227.13:4804 , piece_id=Y7HA6XLUDC2VP7JEKPJNXKHFZJ77U5USNT5RCVW4MS2QZACBOY5Q, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.859311512 (152h29m22.410781973s) [noise=true , address=45.91.116.10:4082 , piece_id=ETWTGCVFPECYXLWSK6U5Q5PAJTKNX5CK2SJJVWCG4KEACZ4IAZOA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.859442551 (152h29m22.410913013s) [noise=true , address=45.91.116.5:4298 , piece_id=GKNZT2GUEIHHPWJQIEH5BSUWO2XXOM6EQ5KZX2JTZSMCNB74FQ4Q, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.861349383 (152h29m22.412819834s) [noise=true , address=45.91.116.10:4083 , piece_id=N7STKXM3QZCERQN6W5VU6K5LMBESIFDANG6MY67XL57TBY5JRK4A, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.861826902 (152h29m22.413297363s) [noise=true , address=145.79.227.31:4799 , piece_id=2X2RCSEPSBRGIQR4TYXYND35QOPWUFWTJMXFXFIYCX3DYMJ442FQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.863192743 (152h29m22.414663204s) [noise=true , address=145.79.227.28:4727 , piece_id=CLFAJR2TX5F2R22PRBV26GTY3FZAM3C3O3AG47CU6N6YRDHODLGQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.864499722 (152h29m22.415970183s) [noise=true , address=145.79.227.11:4732 , piece_id=4W7WUHZE5VJXXYB65GF5QVAGNN4SHLGAD7OES2D4G2JGOYUU2YLQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.864794413 (152h29m22.416264864s) [noise=true , address=145.79.227.26:4653 , piece_id=G64IAIWXOOOHG3M3XMOHXHCXZ2LWSQ2XAB7ALFPAFWKOWEXKYAMQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.866380714 (152h29m22.417851175s) [noise=true , address=45.91.116.4:4940 , piece_id=E6OZVZK4KPEZPVISBIDIR56ATOXKHXWAFRO4KTJ3AN3F5S542ZWQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.867347054 (152h29m22.418817515s) [noise=true , address=145.79.227.15:4873 , piece_id=XDTTIU2YYZ4YX2CR7GFATGW5QV7B6NLO7CP3P7LHE4UFLAFEDLMA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:25.673146503 (152h29m24.224616954s) [upload=134305]: Next returned nil segment Nov 15 16:33:25.673183183 (152h29m24.224653644s) [upload=134305]: Waiting for error group managing segments... Nov 15 16:33:25.957140584 (152h29m24.508611035s) [noise=true , address=45.91.116.10:4083 , piece_id=N7STKXM3QZCERQN6W5VU6K5LMBESIFDANG6MY67XL57TBY5JRK4A, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.001103691 (152h29m24.552574162s) [noise=true , address=45.91.116.4:4940 , piece_id=E6OZVZK4KPEZPVISBIDIR56ATOXKHXWAFRO4KTJ3AN3F5S542ZWQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.003521624 (152h29m24.554992086s) [noise=true , address=45.91.116.5:4298 , piece_id=GKNZT2GUEIHHPWJQIEH5BSUWO2XXOM6EQ5KZX2JTZSMCNB74FQ4Q, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.004722812 (152h29m24.556193293s) [noise=true , address=77.90.31.18:4806 , piece_id=QCWZNPVJANVBAHKV55FW7QYNU27OKAFMP5XDN2TDT7XVLMJMNQQA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.027847948 (152h29m24.579318419s) [noise=true , address=77.90.31.11:4553 , piece_id=M3MGFKZJXBQ5BGYJVLP5HQJ7KGK3BBYULTEGPY4XJQDBAG3T5UIQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.038988902 (152h29m24.590459363s) [noise=true , address=45.91.116.10:4082 , piece_id=ETWTGCVFPECYXLWSK6U5Q5PAJTKNX5CK2SJJVWCG4KEACZ4IAZOA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.049585892 (152h29m24.601056354s) [noise=true , address=45.91.116.14:4190 , piece_id=7KMDRAD6OXDEOZNHHMEWP6IWIB5KBDU6PNA4LPILDN5ZG7QBB6BA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:45.222994074 (152h29m43.774464535s) [noise=true , address=145.79.227.15:4873 , piece_id=XDTTIU2YYZ4YX2CR7GFATGW5QV7B6NLO7CP3P7LHE4UFLAFEDLMA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:52.891596883 (152h29m51.443067334s) [noise=true , address=145.79.227.7:4295 , piece_id=ELJRP3KI6F7ZFOMIYGWSTSAELMSI5H6OS4WU5R5UGA7BC3P3BEGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection Nov 15 16:33:52.891716481 (152h29m51.443186952s) [noise=true , address=145.79.227.8:4329 , piece_id=BZNWOFF4FXQAB4GDDSJLG6ANC4UYY4F743U7E63MCYWEXD6O6R5A, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection Nov 15 16:33:52.891805550 (152h29m51.443276001s) [noise=true , address=145.79.227.11:4732 , piece_id=4W7WUHZE5VJXXYB65GF5QVAGNN4SHLGAD7OES2D4G2JGOYUU2YLQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection Nov 15 16:33:52.891822112 (152h29m51.443292583s) [noise=true , address=145.79.227.31:4799 , piece_id=2X2RCSEPSBRGIQR4TYXYND35QOPWUFWTJMXFXFIYCX3DYMJ442FQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection Nov 15 16:33:52.891924026 (152h29m51.443394477s) [noise=true , address=145.79.227.32:4835 , piece_id=ZFXRT7D575I3NCZEENPTV7T4GEF6LEIBBTY6A5MBQ455ZTRDHYGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection Nov 15 16:33:52.892373372 (152h29m51.443843833s) [noise=true , address=145.79.227.28:4727 , piece_id=CLFAJR2TX5F2R22PRBV26GTY3FZAM3C3O3AG47CU6N6YRDHODLGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection Nov 15 16:33:52.892554417 (152h29m51.444024868s) [noise=true , address=145.79.227.26:4653 , piece_id=G64IAIWXOOOHG3M3XMOHXHCXZ2LWSQ2XAB7ALFPAFWKOWEXKYAMQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection Nov 15 16:33:52.892663404 (152h29m51.444133865s) [noise=true , address=145.79.227.13:4804 , piece_id=Y7HA6XLUDC2VP7JEKPJNXKHFZJ77U5USNT5RCVW4MS2QZACBOY5Q, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection Nov 15 16:33:52.892704532 (152h29m51.444174993s) [seg_pos={0 0}, upload=134305]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 15 16:33:53.136366081 (152h29m51.687836532s) [upload=134305]: Done uploading segments... Nov 15 16:33:53.136405987 (152h29m51.687876448s) [upload=134305]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.827305602s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................X........X...............................] duration: 3.394540458s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:37:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.39456291s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................X........X..............................] duration: 3.394997971s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:36:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.395026534s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X...X....] duration: 5.89386129s trace: https://jaeger.dev.storj.io/trace/5a8562300700bcde FAIL (2025-11-15 16:36:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== e=true , address=145.79.227.8:4329 , piece_id=BZNWOFF4FXQAB4GDDSJLG6ANC4UYY4F743U7E63MCYWEXD6O6R5A, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.851927882 (152h29m22.403398343s) [noise=true , address=45.91.116.14:4190 , piece_id=7KMDRAD6OXDEOZNHHMEWP6IWIB5KBDU6PNA4LPILDN5ZG7QBB6BA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.854137329 (152h29m22.40560779s) [noise=true , address=77.90.31.18:4806 , piece_id=QCWZNPVJANVBAHKV55FW7QYNU27OKAFMP5XDN2TDT7XVLMJMNQQA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.856035504 (152h29m22.407505965s) [noise=true , address=145.79.227.13:4804 , piece_id=Y7HA6XLUDC2VP7JEKPJNXKHFZJ77U5USNT5RCVW4MS2QZACBOY5Q, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.859311512 (152h29m22.410781973s) [noise=true , address=45.91.116.10:4082 , piece_id=ETWTGCVFPECYXLWSK6U5Q5PAJTKNX5CK2SJJVWCG4KEACZ4IAZOA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.859442551 (152h29m22.410913013s) [noise=true , address=45.91.116.5:4298 , piece_id=GKNZT2GUEIHHPWJQIEH5BSUWO2XXOM6EQ5KZX2JTZSMCNB74FQ4Q, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.861349383 (152h29m22.412819834s) [noise=true , address=45.91.116.10:4083 , piece_id=N7STKXM3QZCERQN6W5VU6K5LMBESIFDANG6MY67XL57TBY5JRK4A, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.861826902 (152h29m22.413297363s) [noise=true , address=145.79.227.31:4799 , piece_id=2X2RCSEPSBRGIQR4TYXYND35QOPWUFWTJMXFXFIYCX3DYMJ442FQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.863192743 (152h29m22.414663204s) [noise=true , address=145.79.227.28:4727 , piece_id=CLFAJR2TX5F2R22PRBV26GTY3FZAM3C3O3AG47CU6N6YRDHODLGQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.864499722 (152h29m22.415970183s) [noise=true , address=145.79.227.11:4732 , piece_id=4W7WUHZE5VJXXYB65GF5QVAGNN4SHLGAD7OES2D4G2JGOYUU2YLQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.864794413 (152h29m22.416264864s) [noise=true , address=145.79.227.26:4653 , piece_id=G64IAIWXOOOHG3M3XMOHXHCXZ2LWSQ2XAB7ALFPAFWKOWEXKYAMQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.866380714 (152h29m22.417851175s) [noise=true , address=45.91.116.4:4940 , piece_id=E6OZVZK4KPEZPVISBIDIR56ATOXKHXWAFRO4KTJ3AN3F5S542ZWQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.867347054 (152h29m22.418817515s) [noise=true , address=145.79.227.15:4873 , piece_id=XDTTIU2YYZ4YX2CR7GFATGW5QV7B6NLO7CP3P7LHE4UFLAFEDLMA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:25.673146503 (152h29m24.224616954s) [upload=134305]: Next returned nil segment Nov 15 16:33:25.673183183 (152h29m24.224653644s) [upload=134305]: Waiting for error group managing segments... Nov 15 16:33:25.957140584 (152h29m24.508611035s) [noise=true , address=45.91.116.10:4083 , piece_id=N7STKXM3QZCERQN6W5VU6K5LMBESIFDANG6MY67XL57TBY5JRK4A, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.001103691 (152h29m24.552574162s) [noise=true , address=45.91.116.4:4940 , piece_id=E6OZVZK4KPEZPVISBIDIR56ATOXKHXWAFRO4KTJ3AN3F5S542ZWQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.003521624 (152h29m24.554992086s) [noise=true , address=45.91.116.5:4298 , piece_id=GKNZT2GUEIHHPWJQIEH5BSUWO2XXOM6EQ5KZX2JTZSMCNB74FQ4Q, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.004722812 (152h29m24.556193293s) [noise=true , address=77.90.31.18:4806 , piece_id=QCWZNPVJANVBAHKV55FW7QYNU27OKAFMP5XDN2TDT7XVLMJMNQQA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.027847948 (152h29m24.579318419s) [noise=true , address=77.90.31.11:4553 , piece_id=M3MGFKZJXBQ5BGYJVLP5HQJ7KGK3BBYULTEGPY4XJQDBAG3T5UIQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.038988902 (152h29m24.590459363s) [noise=true , address=45.91.116.10:4082 , piece_id=ETWTGCVFPECYXLWSK6U5Q5PAJTKNX5CK2SJJVWCG4KEACZ4IAZOA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.049585892 (152h29m24.601056354s) [noise=true , address=45.91.116.14:4190 , piece_id=7KMDRAD6OXDEOZNHHMEWP6IWIB5KBDU6PNA4LPILDN5ZG7QBB6BA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:45.222994074 (152h29m43.774464535s) [noise=true , address=145.79.227.15:4873 , piece_id=XDTTIU2YYZ4YX2CR7GFATGW5QV7B6NLO7CP3P7LHE4UFLAFEDLMA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:52.891596883 (152h29m51.443067334s) [noise=true , address=145.79.227.7:4295 , piece_id=ELJRP3KI6F7ZFOMIYGWSTSAELMSI5H6OS4WU5R5UGA7BC3P3BEGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection Nov 15 16:33:52.891716481 (152h29m51.443186952s) [noise=true , address=145.79.227.8:4329 , piece_id=BZNWOFF4FXQAB4GDDSJLG6ANC4UYY4F743U7E63MCYWEXD6O6R5A, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection Nov 15 16:33:52.891805550 (152h29m51.443276001s) [noise=true , address=145.79.227.11:4732 , piece_id=4W7WUHZE5VJXXYB65GF5QVAGNN4SHLGAD7OES2D4G2JGOYUU2YLQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection Nov 15 16:33:52.891822112 (152h29m51.443292583s) [noise=true , address=145.79.227.31:4799 , piece_id=2X2RCSEPSBRGIQR4TYXYND35QOPWUFWTJMXFXFIYCX3DYMJ442FQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection Nov 15 16:33:52.891924026 (152h29m51.443394477s) [noise=true , address=145.79.227.32:4835 , piece_id=ZFXRT7D575I3NCZEENPTV7T4GEF6LEIBBTY6A5MBQ455ZTRDHYGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection Nov 15 16:33:52.892373372 (152h29m51.443843833s) [noise=true , address=145.79.227.28:4727 , piece_id=CLFAJR2TX5F2R22PRBV26GTY3FZAM3C3O3AG47CU6N6YRDHODLGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection Nov 15 16:33:52.892554417 (152h29m51.444024868s) [noise=true , address=145.79.227.26:4653 , piece_id=G64IAIWXOOOHG3M3XMOHXHCXZ2LWSQ2XAB7ALFPAFWKOWEXKYAMQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection Nov 15 16:33:52.892663404 (152h29m51.444133865s) [noise=true , address=145.79.227.13:4804 , piece_id=Y7HA6XLUDC2VP7JEKPJNXKHFZJ77U5USNT5RCVW4MS2QZACBOY5Q, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection Nov 15 16:33:52.892704532 (152h29m51.444174993s) [seg_pos={0 0}, upload=134305]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 15 16:33:53.136366081 (152h29m51.687836532s) [upload=134305]: Done uploading segments... Nov 15 16:33:53.136405987 (152h29m51.687876448s) [upload=134305]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.893895394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X...X...] duration: 6.071299086s trace: https://jaeger.dev.storj.io/trace/5a8562300700bcde FAIL (2025-11-15 16:35:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== e=true , address=145.79.227.8:4329 , piece_id=BZNWOFF4FXQAB4GDDSJLG6ANC4UYY4F743U7E63MCYWEXD6O6R5A, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.851927882 (152h29m22.403398343s) [noise=true , address=45.91.116.14:4190 , piece_id=7KMDRAD6OXDEOZNHHMEWP6IWIB5KBDU6PNA4LPILDN5ZG7QBB6BA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.854137329 (152h29m22.40560779s) [noise=true , address=77.90.31.18:4806 , piece_id=QCWZNPVJANVBAHKV55FW7QYNU27OKAFMP5XDN2TDT7XVLMJMNQQA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.856035504 (152h29m22.407505965s) [noise=true , address=145.79.227.13:4804 , piece_id=Y7HA6XLUDC2VP7JEKPJNXKHFZJ77U5USNT5RCVW4MS2QZACBOY5Q, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.859311512 (152h29m22.410781973s) [noise=true , address=45.91.116.10:4082 , piece_id=ETWTGCVFPECYXLWSK6U5Q5PAJTKNX5CK2SJJVWCG4KEACZ4IAZOA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.859442551 (152h29m22.410913013s) [noise=true , address=45.91.116.5:4298 , piece_id=GKNZT2GUEIHHPWJQIEH5BSUWO2XXOM6EQ5KZX2JTZSMCNB74FQ4Q, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.861349383 (152h29m22.412819834s) [noise=true , address=45.91.116.10:4083 , piece_id=N7STKXM3QZCERQN6W5VU6K5LMBESIFDANG6MY67XL57TBY5JRK4A, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.861826902 (152h29m22.413297363s) [noise=true , address=145.79.227.31:4799 , piece_id=2X2RCSEPSBRGIQR4TYXYND35QOPWUFWTJMXFXFIYCX3DYMJ442FQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.863192743 (152h29m22.414663204s) [noise=true , address=145.79.227.28:4727 , piece_id=CLFAJR2TX5F2R22PRBV26GTY3FZAM3C3O3AG47CU6N6YRDHODLGQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.864499722 (152h29m22.415970183s) [noise=true , address=145.79.227.11:4732 , piece_id=4W7WUHZE5VJXXYB65GF5QVAGNN4SHLGAD7OES2D4G2JGOYUU2YLQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.864794413 (152h29m22.416264864s) [noise=true , address=145.79.227.26:4653 , piece_id=G64IAIWXOOOHG3M3XMOHXHCXZ2LWSQ2XAB7ALFPAFWKOWEXKYAMQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.866380714 (152h29m22.417851175s) [noise=true , address=45.91.116.4:4940 , piece_id=E6OZVZK4KPEZPVISBIDIR56ATOXKHXWAFRO4KTJ3AN3F5S542ZWQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.867347054 (152h29m22.418817515s) [noise=true , address=145.79.227.15:4873 , piece_id=XDTTIU2YYZ4YX2CR7GFATGW5QV7B6NLO7CP3P7LHE4UFLAFEDLMA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:25.673146503 (152h29m24.224616954s) [upload=134305]: Next returned nil segment Nov 15 16:33:25.673183183 (152h29m24.224653644s) [upload=134305]: Waiting for error group managing segments... Nov 15 16:33:25.957140584 (152h29m24.508611035s) [noise=true , address=45.91.116.10:4083 , piece_id=N7STKXM3QZCERQN6W5VU6K5LMBESIFDANG6MY67XL57TBY5JRK4A, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.001103691 (152h29m24.552574162s) [noise=true , address=45.91.116.4:4940 , piece_id=E6OZVZK4KPEZPVISBIDIR56ATOXKHXWAFRO4KTJ3AN3F5S542ZWQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.003521624 (152h29m24.554992086s) [noise=true , address=45.91.116.5:4298 , piece_id=GKNZT2GUEIHHPWJQIEH5BSUWO2XXOM6EQ5KZX2JTZSMCNB74FQ4Q, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.004722812 (152h29m24.556193293s) [noise=true , address=77.90.31.18:4806 , piece_id=QCWZNPVJANVBAHKV55FW7QYNU27OKAFMP5XDN2TDT7XVLMJMNQQA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.027847948 (152h29m24.579318419s) [noise=true , address=77.90.31.11:4553 , piece_id=M3MGFKZJXBQ5BGYJVLP5HQJ7KGK3BBYULTEGPY4XJQDBAG3T5UIQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.038988902 (152h29m24.590459363s) [noise=true , address=45.91.116.10:4082 , piece_id=ETWTGCVFPECYXLWSK6U5Q5PAJTKNX5CK2SJJVWCG4KEACZ4IAZOA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.049585892 (152h29m24.601056354s) [noise=true , address=45.91.116.14:4190 , piece_id=7KMDRAD6OXDEOZNHHMEWP6IWIB5KBDU6PNA4LPILDN5ZG7QBB6BA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:45.222994074 (152h29m43.774464535s) [noise=true , address=145.79.227.15:4873 , piece_id=XDTTIU2YYZ4YX2CR7GFATGW5QV7B6NLO7CP3P7LHE4UFLAFEDLMA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:52.891596883 (152h29m51.443067334s) [noise=true , address=145.79.227.7:4295 , piece_id=ELJRP3KI6F7ZFOMIYGWSTSAELMSI5H6OS4WU5R5UGA7BC3P3BEGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection Nov 15 16:33:52.891716481 (152h29m51.443186952s) [noise=true , address=145.79.227.8:4329 , piece_id=BZNWOFF4FXQAB4GDDSJLG6ANC4UYY4F743U7E63MCYWEXD6O6R5A, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection Nov 15 16:33:52.891805550 (152h29m51.443276001s) [noise=true , address=145.79.227.11:4732 , piece_id=4W7WUHZE5VJXXYB65GF5QVAGNN4SHLGAD7OES2D4G2JGOYUU2YLQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection Nov 15 16:33:52.891822112 (152h29m51.443292583s) [noise=true , address=145.79.227.31:4799 , piece_id=2X2RCSEPSBRGIQR4TYXYND35QOPWUFWTJMXFXFIYCX3DYMJ442FQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection Nov 15 16:33:52.891924026 (152h29m51.443394477s) [noise=true , address=145.79.227.32:4835 , piece_id=ZFXRT7D575I3NCZEENPTV7T4GEF6LEIBBTY6A5MBQ455ZTRDHYGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection Nov 15 16:33:52.892373372 (152h29m51.443843833s) [noise=true , address=145.79.227.28:4727 , piece_id=CLFAJR2TX5F2R22PRBV26GTY3FZAM3C3O3AG47CU6N6YRDHODLGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection Nov 15 16:33:52.892554417 (152h29m51.444024868s) [noise=true , address=145.79.227.26:4653 , piece_id=G64IAIWXOOOHG3M3XMOHXHCXZ2LWSQ2XAB7ALFPAFWKOWEXKYAMQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection Nov 15 16:33:52.892663404 (152h29m51.444133865s) [noise=true , address=145.79.227.13:4804 , piece_id=Y7HA6XLUDC2VP7JEKPJNXKHFZJ77U5USNT5RCVW4MS2QZACBOY5Q, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection Nov 15 16:33:52.892704532 (152h29m51.444174993s) [seg_pos={0 0}, upload=134305]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 15 16:33:53.136366081 (152h29m51.687836532s) [upload=134305]: Done uploading segments... Nov 15 16:33:53.136405987 (152h29m51.687876448s) [upload=134305]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.071319245s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................X........X.............................] duration: 3.393071277s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:35:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.393090293s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X...X..] duration: 6.055929991s trace: https://jaeger.dev.storj.io/trace/5a8562300700bcde FAIL (2025-11-15 16:35:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== e=true , address=145.79.227.8:4329 , piece_id=BZNWOFF4FXQAB4GDDSJLG6ANC4UYY4F743U7E63MCYWEXD6O6R5A, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.851927882 (152h29m22.403398343s) [noise=true , address=45.91.116.14:4190 , piece_id=7KMDRAD6OXDEOZNHHMEWP6IWIB5KBDU6PNA4LPILDN5ZG7QBB6BA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.854137329 (152h29m22.40560779s) [noise=true , address=77.90.31.18:4806 , piece_id=QCWZNPVJANVBAHKV55FW7QYNU27OKAFMP5XDN2TDT7XVLMJMNQQA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.856035504 (152h29m22.407505965s) [noise=true , address=145.79.227.13:4804 , piece_id=Y7HA6XLUDC2VP7JEKPJNXKHFZJ77U5USNT5RCVW4MS2QZACBOY5Q, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.859311512 (152h29m22.410781973s) [noise=true , address=45.91.116.10:4082 , piece_id=ETWTGCVFPECYXLWSK6U5Q5PAJTKNX5CK2SJJVWCG4KEACZ4IAZOA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.859442551 (152h29m22.410913013s) [noise=true , address=45.91.116.5:4298 , piece_id=GKNZT2GUEIHHPWJQIEH5BSUWO2XXOM6EQ5KZX2JTZSMCNB74FQ4Q, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.861349383 (152h29m22.412819834s) [noise=true , address=45.91.116.10:4083 , piece_id=N7STKXM3QZCERQN6W5VU6K5LMBESIFDANG6MY67XL57TBY5JRK4A, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.861826902 (152h29m22.413297363s) [noise=true , address=145.79.227.31:4799 , piece_id=2X2RCSEPSBRGIQR4TYXYND35QOPWUFWTJMXFXFIYCX3DYMJ442FQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.863192743 (152h29m22.414663204s) [noise=true , address=145.79.227.28:4727 , piece_id=CLFAJR2TX5F2R22PRBV26GTY3FZAM3C3O3AG47CU6N6YRDHODLGQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.864499722 (152h29m22.415970183s) [noise=true , address=145.79.227.11:4732 , piece_id=4W7WUHZE5VJXXYB65GF5QVAGNN4SHLGAD7OES2D4G2JGOYUU2YLQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.864794413 (152h29m22.416264864s) [noise=true , address=145.79.227.26:4653 , piece_id=G64IAIWXOOOHG3M3XMOHXHCXZ2LWSQ2XAB7ALFPAFWKOWEXKYAMQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.866380714 (152h29m22.417851175s) [noise=true , address=45.91.116.4:4940 , piece_id=E6OZVZK4KPEZPVISBIDIR56ATOXKHXWAFRO4KTJ3AN3F5S542ZWQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.867347054 (152h29m22.418817515s) [noise=true , address=145.79.227.15:4873 , piece_id=XDTTIU2YYZ4YX2CR7GFATGW5QV7B6NLO7CP3P7LHE4UFLAFEDLMA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:25.673146503 (152h29m24.224616954s) [upload=134305]: Next returned nil segment Nov 15 16:33:25.673183183 (152h29m24.224653644s) [upload=134305]: Waiting for error group managing segments... Nov 15 16:33:25.957140584 (152h29m24.508611035s) [noise=true , address=45.91.116.10:4083 , piece_id=N7STKXM3QZCERQN6W5VU6K5LMBESIFDANG6MY67XL57TBY5JRK4A, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.001103691 (152h29m24.552574162s) [noise=true , address=45.91.116.4:4940 , piece_id=E6OZVZK4KPEZPVISBIDIR56ATOXKHXWAFRO4KTJ3AN3F5S542ZWQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.003521624 (152h29m24.554992086s) [noise=true , address=45.91.116.5:4298 , piece_id=GKNZT2GUEIHHPWJQIEH5BSUWO2XXOM6EQ5KZX2JTZSMCNB74FQ4Q, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.004722812 (152h29m24.556193293s) [noise=true , address=77.90.31.18:4806 , piece_id=QCWZNPVJANVBAHKV55FW7QYNU27OKAFMP5XDN2TDT7XVLMJMNQQA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.027847948 (152h29m24.579318419s) [noise=true , address=77.90.31.11:4553 , piece_id=M3MGFKZJXBQ5BGYJVLP5HQJ7KGK3BBYULTEGPY4XJQDBAG3T5UIQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.038988902 (152h29m24.590459363s) [noise=true , address=45.91.116.10:4082 , piece_id=ETWTGCVFPECYXLWSK6U5Q5PAJTKNX5CK2SJJVWCG4KEACZ4IAZOA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.049585892 (152h29m24.601056354s) [noise=true , address=45.91.116.14:4190 , piece_id=7KMDRAD6OXDEOZNHHMEWP6IWIB5KBDU6PNA4LPILDN5ZG7QBB6BA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:45.222994074 (152h29m43.774464535s) [noise=true , address=145.79.227.15:4873 , piece_id=XDTTIU2YYZ4YX2CR7GFATGW5QV7B6NLO7CP3P7LHE4UFLAFEDLMA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:52.891596883 (152h29m51.443067334s) [noise=true , address=145.79.227.7:4295 , piece_id=ELJRP3KI6F7ZFOMIYGWSTSAELMSI5H6OS4WU5R5UGA7BC3P3BEGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection Nov 15 16:33:52.891716481 (152h29m51.443186952s) [noise=true , address=145.79.227.8:4329 , piece_id=BZNWOFF4FXQAB4GDDSJLG6ANC4UYY4F743U7E63MCYWEXD6O6R5A, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection Nov 15 16:33:52.891805550 (152h29m51.443276001s) [noise=true , address=145.79.227.11:4732 , piece_id=4W7WUHZE5VJXXYB65GF5QVAGNN4SHLGAD7OES2D4G2JGOYUU2YLQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection Nov 15 16:33:52.891822112 (152h29m51.443292583s) [noise=true , address=145.79.227.31:4799 , piece_id=2X2RCSEPSBRGIQR4TYXYND35QOPWUFWTJMXFXFIYCX3DYMJ442FQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection Nov 15 16:33:52.891924026 (152h29m51.443394477s) [noise=true , address=145.79.227.32:4835 , piece_id=ZFXRT7D575I3NCZEENPTV7T4GEF6LEIBBTY6A5MBQ455ZTRDHYGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection Nov 15 16:33:52.892373372 (152h29m51.443843833s) [noise=true , address=145.79.227.28:4727 , piece_id=CLFAJR2TX5F2R22PRBV26GTY3FZAM3C3O3AG47CU6N6YRDHODLGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection Nov 15 16:33:52.892554417 (152h29m51.444024868s) [noise=true , address=145.79.227.26:4653 , piece_id=G64IAIWXOOOHG3M3XMOHXHCXZ2LWSQ2XAB7ALFPAFWKOWEXKYAMQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection Nov 15 16:33:52.892663404 (152h29m51.444133865s) [noise=true , address=145.79.227.13:4804 , piece_id=Y7HA6XLUDC2VP7JEKPJNXKHFZJ77U5USNT5RCVW4MS2QZACBOY5Q, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection Nov 15 16:33:52.892704532 (152h29m51.444174993s) [seg_pos={0 0}, upload=134305]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 15 16:33:53.136366081 (152h29m51.687836532s) [upload=134305]: Done uploading segments... Nov 15 16:33:53.136405987 (152h29m51.687876448s) [upload=134305]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.055955709s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X...X.] duration: 6.330974431s trace: https://jaeger.dev.storj.io/trace/5a8562300700bcde FAIL (2025-11-15 16:34:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== e=true , address=145.79.227.8:4329 , piece_id=BZNWOFF4FXQAB4GDDSJLG6ANC4UYY4F743U7E63MCYWEXD6O6R5A, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.851927882 (152h29m22.403398343s) [noise=true , address=45.91.116.14:4190 , piece_id=7KMDRAD6OXDEOZNHHMEWP6IWIB5KBDU6PNA4LPILDN5ZG7QBB6BA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.854137329 (152h29m22.40560779s) [noise=true , address=77.90.31.18:4806 , piece_id=QCWZNPVJANVBAHKV55FW7QYNU27OKAFMP5XDN2TDT7XVLMJMNQQA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.856035504 (152h29m22.407505965s) [noise=true , address=145.79.227.13:4804 , piece_id=Y7HA6XLUDC2VP7JEKPJNXKHFZJ77U5USNT5RCVW4MS2QZACBOY5Q, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.859311512 (152h29m22.410781973s) [noise=true , address=45.91.116.10:4082 , piece_id=ETWTGCVFPECYXLWSK6U5Q5PAJTKNX5CK2SJJVWCG4KEACZ4IAZOA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.859442551 (152h29m22.410913013s) [noise=true , address=45.91.116.5:4298 , piece_id=GKNZT2GUEIHHPWJQIEH5BSUWO2XXOM6EQ5KZX2JTZSMCNB74FQ4Q, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.861349383 (152h29m22.412819834s) [noise=true , address=45.91.116.10:4083 , piece_id=N7STKXM3QZCERQN6W5VU6K5LMBESIFDANG6MY67XL57TBY5JRK4A, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.861826902 (152h29m22.413297363s) [noise=true , address=145.79.227.31:4799 , piece_id=2X2RCSEPSBRGIQR4TYXYND35QOPWUFWTJMXFXFIYCX3DYMJ442FQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.863192743 (152h29m22.414663204s) [noise=true , address=145.79.227.28:4727 , piece_id=CLFAJR2TX5F2R22PRBV26GTY3FZAM3C3O3AG47CU6N6YRDHODLGQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.864499722 (152h29m22.415970183s) [noise=true , address=145.79.227.11:4732 , piece_id=4W7WUHZE5VJXXYB65GF5QVAGNN4SHLGAD7OES2D4G2JGOYUU2YLQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.864794413 (152h29m22.416264864s) [noise=true , address=145.79.227.26:4653 , piece_id=G64IAIWXOOOHG3M3XMOHXHCXZ2LWSQ2XAB7ALFPAFWKOWEXKYAMQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.866380714 (152h29m22.417851175s) [noise=true , address=45.91.116.4:4940 , piece_id=E6OZVZK4KPEZPVISBIDIR56ATOXKHXWAFRO4KTJ3AN3F5S542ZWQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.867347054 (152h29m22.418817515s) [noise=true , address=145.79.227.15:4873 , piece_id=XDTTIU2YYZ4YX2CR7GFATGW5QV7B6NLO7CP3P7LHE4UFLAFEDLMA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:25.673146503 (152h29m24.224616954s) [upload=134305]: Next returned nil segment Nov 15 16:33:25.673183183 (152h29m24.224653644s) [upload=134305]: Waiting for error group managing segments... Nov 15 16:33:25.957140584 (152h29m24.508611035s) [noise=true , address=45.91.116.10:4083 , piece_id=N7STKXM3QZCERQN6W5VU6K5LMBESIFDANG6MY67XL57TBY5JRK4A, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.001103691 (152h29m24.552574162s) [noise=true , address=45.91.116.4:4940 , piece_id=E6OZVZK4KPEZPVISBIDIR56ATOXKHXWAFRO4KTJ3AN3F5S542ZWQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.003521624 (152h29m24.554992086s) [noise=true , address=45.91.116.5:4298 , piece_id=GKNZT2GUEIHHPWJQIEH5BSUWO2XXOM6EQ5KZX2JTZSMCNB74FQ4Q, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.004722812 (152h29m24.556193293s) [noise=true , address=77.90.31.18:4806 , piece_id=QCWZNPVJANVBAHKV55FW7QYNU27OKAFMP5XDN2TDT7XVLMJMNQQA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.027847948 (152h29m24.579318419s) [noise=true , address=77.90.31.11:4553 , piece_id=M3MGFKZJXBQ5BGYJVLP5HQJ7KGK3BBYULTEGPY4XJQDBAG3T5UIQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.038988902 (152h29m24.590459363s) [noise=true , address=45.91.116.10:4082 , piece_id=ETWTGCVFPECYXLWSK6U5Q5PAJTKNX5CK2SJJVWCG4KEACZ4IAZOA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.049585892 (152h29m24.601056354s) [noise=true , address=45.91.116.14:4190 , piece_id=7KMDRAD6OXDEOZNHHMEWP6IWIB5KBDU6PNA4LPILDN5ZG7QBB6BA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:45.222994074 (152h29m43.774464535s) [noise=true , address=145.79.227.15:4873 , piece_id=XDTTIU2YYZ4YX2CR7GFATGW5QV7B6NLO7CP3P7LHE4UFLAFEDLMA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:52.891596883 (152h29m51.443067334s) [noise=true , address=145.79.227.7:4295 , piece_id=ELJRP3KI6F7ZFOMIYGWSTSAELMSI5H6OS4WU5R5UGA7BC3P3BEGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection Nov 15 16:33:52.891716481 (152h29m51.443186952s) [noise=true , address=145.79.227.8:4329 , piece_id=BZNWOFF4FXQAB4GDDSJLG6ANC4UYY4F743U7E63MCYWEXD6O6R5A, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection Nov 15 16:33:52.891805550 (152h29m51.443276001s) [noise=true , address=145.79.227.11:4732 , piece_id=4W7WUHZE5VJXXYB65GF5QVAGNN4SHLGAD7OES2D4G2JGOYUU2YLQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection Nov 15 16:33:52.891822112 (152h29m51.443292583s) [noise=true , address=145.79.227.31:4799 , piece_id=2X2RCSEPSBRGIQR4TYXYND35QOPWUFWTJMXFXFIYCX3DYMJ442FQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection Nov 15 16:33:52.891924026 (152h29m51.443394477s) [noise=true , address=145.79.227.32:4835 , piece_id=ZFXRT7D575I3NCZEENPTV7T4GEF6LEIBBTY6A5MBQ455ZTRDHYGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection Nov 15 16:33:52.892373372 (152h29m51.443843833s) [noise=true , address=145.79.227.28:4727 , piece_id=CLFAJR2TX5F2R22PRBV26GTY3FZAM3C3O3AG47CU6N6YRDHODLGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection Nov 15 16:33:52.892554417 (152h29m51.444024868s) [noise=true , address=145.79.227.26:4653 , piece_id=G64IAIWXOOOHG3M3XMOHXHCXZ2LWSQ2XAB7ALFPAFWKOWEXKYAMQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection Nov 15 16:33:52.892663404 (152h29m51.444133865s) [noise=true , address=145.79.227.13:4804 , piece_id=Y7HA6XLUDC2VP7JEKPJNXKHFZJ77U5USNT5RCVW4MS2QZACBOY5Q, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection Nov 15 16:33:52.892704532 (152h29m51.444174993s) [seg_pos={0 0}, upload=134305]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 15 16:33:53.136366081 (152h29m51.687836532s) [upload=134305]: Done uploading segments... Nov 15 16:33:53.136405987 (152h29m51.687876448s) [upload=134305]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.331002134s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................X........X............................] duration: 3.493941416s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:34:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.49397003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X...X] duration: 30.247031765s trace: https://jaeger.dev.storj.io/trace/5a8562300700bcde FAIL (2025-11-15 16:33:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== e=true , address=145.79.227.8:4329 , piece_id=BZNWOFF4FXQAB4GDDSJLG6ANC4UYY4F743U7E63MCYWEXD6O6R5A, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.851927882 (152h29m22.403398343s) [noise=true , address=45.91.116.14:4190 , piece_id=7KMDRAD6OXDEOZNHHMEWP6IWIB5KBDU6PNA4LPILDN5ZG7QBB6BA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.854137329 (152h29m22.40560779s) [noise=true , address=77.90.31.18:4806 , piece_id=QCWZNPVJANVBAHKV55FW7QYNU27OKAFMP5XDN2TDT7XVLMJMNQQA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.856035504 (152h29m22.407505965s) [noise=true , address=145.79.227.13:4804 , piece_id=Y7HA6XLUDC2VP7JEKPJNXKHFZJ77U5USNT5RCVW4MS2QZACBOY5Q, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.859311512 (152h29m22.410781973s) [noise=true , address=45.91.116.10:4082 , piece_id=ETWTGCVFPECYXLWSK6U5Q5PAJTKNX5CK2SJJVWCG4KEACZ4IAZOA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.859442551 (152h29m22.410913013s) [noise=true , address=45.91.116.5:4298 , piece_id=GKNZT2GUEIHHPWJQIEH5BSUWO2XXOM6EQ5KZX2JTZSMCNB74FQ4Q, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.861349383 (152h29m22.412819834s) [noise=true , address=45.91.116.10:4083 , piece_id=N7STKXM3QZCERQN6W5VU6K5LMBESIFDANG6MY67XL57TBY5JRK4A, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.861826902 (152h29m22.413297363s) [noise=true , address=145.79.227.31:4799 , piece_id=2X2RCSEPSBRGIQR4TYXYND35QOPWUFWTJMXFXFIYCX3DYMJ442FQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.863192743 (152h29m22.414663204s) [noise=true , address=145.79.227.28:4727 , piece_id=CLFAJR2TX5F2R22PRBV26GTY3FZAM3C3O3AG47CU6N6YRDHODLGQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.864499722 (152h29m22.415970183s) [noise=true , address=145.79.227.11:4732 , piece_id=4W7WUHZE5VJXXYB65GF5QVAGNN4SHLGAD7OES2D4G2JGOYUU2YLQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.864794413 (152h29m22.416264864s) [noise=true , address=145.79.227.26:4653 , piece_id=G64IAIWXOOOHG3M3XMOHXHCXZ2LWSQ2XAB7ALFPAFWKOWEXKYAMQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.866380714 (152h29m22.417851175s) [noise=true , address=45.91.116.4:4940 , piece_id=E6OZVZK4KPEZPVISBIDIR56ATOXKHXWAFRO4KTJ3AN3F5S542ZWQ, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:23.867347054 (152h29m22.418817515s) [noise=true , address=145.79.227.15:4873 , piece_id=XDTTIU2YYZ4YX2CR7GFATGW5QV7B6NLO7CP3P7LHE4UFLAFEDLMA, seg_pos={0 0}, upload=134305]: Uploading piece... Nov 15 16:33:25.673146503 (152h29m24.224616954s) [upload=134305]: Next returned nil segment Nov 15 16:33:25.673183183 (152h29m24.224653644s) [upload=134305]: Waiting for error group managing segments... Nov 15 16:33:25.957140584 (152h29m24.508611035s) [noise=true , address=45.91.116.10:4083 , piece_id=N7STKXM3QZCERQN6W5VU6K5LMBESIFDANG6MY67XL57TBY5JRK4A, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.001103691 (152h29m24.552574162s) [noise=true , address=45.91.116.4:4940 , piece_id=E6OZVZK4KPEZPVISBIDIR56ATOXKHXWAFRO4KTJ3AN3F5S542ZWQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.003521624 (152h29m24.554992086s) [noise=true , address=45.91.116.5:4298 , piece_id=GKNZT2GUEIHHPWJQIEH5BSUWO2XXOM6EQ5KZX2JTZSMCNB74FQ4Q, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.004722812 (152h29m24.556193293s) [noise=true , address=77.90.31.18:4806 , piece_id=QCWZNPVJANVBAHKV55FW7QYNU27OKAFMP5XDN2TDT7XVLMJMNQQA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.027847948 (152h29m24.579318419s) [noise=true , address=77.90.31.11:4553 , piece_id=M3MGFKZJXBQ5BGYJVLP5HQJ7KGK3BBYULTEGPY4XJQDBAG3T5UIQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.038988902 (152h29m24.590459363s) [noise=true , address=45.91.116.10:4082 , piece_id=ETWTGCVFPECYXLWSK6U5Q5PAJTKNX5CK2SJJVWCG4KEACZ4IAZOA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:26.049585892 (152h29m24.601056354s) [noise=true , address=45.91.116.14:4190 , piece_id=7KMDRAD6OXDEOZNHHMEWP6IWIB5KBDU6PNA4LPILDN5ZG7QBB6BA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:45.222994074 (152h29m43.774464535s) [noise=true , address=145.79.227.15:4873 , piece_id=XDTTIU2YYZ4YX2CR7GFATGW5QV7B6NLO7CP3P7LHE4UFLAFEDLMA, seg_pos={0 0}, upload=134305]: Done uploading piece. err: <nil> Nov 15 16:33:52.891596883 (152h29m51.443067334s) [noise=true , address=145.79.227.7:4295 , piece_id=ELJRP3KI6F7ZFOMIYGWSTSAELMSI5H6OS4WU5R5UGA7BC3P3BEGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50756->145.79.227.7:4295: use of closed network connection Nov 15 16:33:52.891716481 (152h29m51.443186952s) [noise=true , address=145.79.227.8:4329 , piece_id=BZNWOFF4FXQAB4GDDSJLG6ANC4UYY4F743U7E63MCYWEXD6O6R5A, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59468->145.79.227.8:4329: use of closed network connection Nov 15 16:33:52.891805550 (152h29m51.443276001s) [noise=true , address=145.79.227.11:4732 , piece_id=4W7WUHZE5VJXXYB65GF5QVAGNN4SHLGAD7OES2D4G2JGOYUU2YLQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39072->145.79.227.11:4732: use of closed network connection Nov 15 16:33:52.891822112 (152h29m51.443292583s) [noise=true , address=145.79.227.31:4799 , piece_id=2X2RCSEPSBRGIQR4TYXYND35QOPWUFWTJMXFXFIYCX3DYMJ442FQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52588->145.79.227.31:4799: use of closed network connection Nov 15 16:33:52.891924026 (152h29m51.443394477s) [noise=true , address=145.79.227.32:4835 , piece_id=ZFXRT7D575I3NCZEENPTV7T4GEF6LEIBBTY6A5MBQ455ZTRDHYGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48718->145.79.227.32:4835: use of closed network connection Nov 15 16:33:52.892373372 (152h29m51.443843833s) [noise=true , address=145.79.227.28:4727 , piece_id=CLFAJR2TX5F2R22PRBV26GTY3FZAM3C3O3AG47CU6N6YRDHODLGQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48154->145.79.227.28:4727: use of closed network connection Nov 15 16:33:52.892554417 (152h29m51.444024868s) [noise=true , address=145.79.227.26:4653 , piece_id=G64IAIWXOOOHG3M3XMOHXHCXZ2LWSQ2XAB7ALFPAFWKOWEXKYAMQ, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50520->145.79.227.26:4653: use of closed network connection Nov 15 16:33:52.892663404 (152h29m51.444133865s) [noise=true , address=145.79.227.13:4804 , piece_id=Y7HA6XLUDC2VP7JEKPJNXKHFZJ77U5USNT5RCVW4MS2QZACBOY5Q, seg_pos={0 0}, upload=134305]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52558->145.79.227.13:4804: use of closed network connection Nov 15 16:33:52.892704532 (152h29m51.444174993s) [seg_pos={0 0}, upload=134305]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 15 16:33:53.136366081 (152h29m51.687836532s) [upload=134305]: Done uploading segments... Nov 15 16:33:53.136405987 (152h29m51.687876448s) [upload=134305]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.247057624s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................X........X...........................] duration: 3.492222106s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:33:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.492300714s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................X........X..........................] duration: 3.485009219s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:32:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.485039176s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.077085844s trace: https://jaeger.dev.storj.io/trace/6bf6128111f84872 FAIL (2025-11-15 16:32:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HK62ESSQE4Q4AA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.095222666 (152h26m31.646693117s) [noise=true , address=145.79.227.23:4546 , piece_id=D6U2YPSDJMP3CDRRXMHWKFHAB4MQMEDJ75F7LKDNZ5URMOIKE5UA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.095747606 (152h26m31.647218067s) [noise=true , address=145.79.227.10:4402 , piece_id=DG4WNF766BPUCG7LTYYDMYDVGRJU66T2LQU7NHMGHZMQO24652YA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.097728758 (152h26m31.649199219s) [noise=true , address=145.79.227.15:4874 , piece_id=2BVYQHWPNLMNVFY6YBPCPYCIJ4V366CPQ4ZBORU5ES6O25ZKCISA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.098417129 (152h26m31.64988758s) [noise=true , address=45.91.116.11:4117 , piece_id=EWRHIJQXTRBQ2HJEXZ26AA23QNKNX5T5YXTLV73G7JDRU26FJGZQ, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.100496519 (152h26m31.65196698s) [noise=true , address=145.79.227.32:4836 , piece_id=VI4GQZITGKW6ONCICLUC3IYHZTJMS2PNC2N53KA6F3VQMTHA3WXQ, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.102319971 (152h26m31.653790432s) [noise=true , address=145.79.227.9:4365 , piece_id=GG6FPM3K3IGM5HBGUEISASLYQID3AI4RCRQT3LX2SRVUGYKYTWLQ, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.103883157 (152h26m31.655353619s) [noise=true , address=145.79.227.17:4970 , piece_id=SBE5DL43YQF5H762LPMHPWJKZTSWJDLPWDU6JQW7GROYTZZXGYLQ, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.105434873 (152h26m31.656905334s) [noise=true , address=77.90.31.19:4841 , piece_id=N6LBPIHIVTBI2S4GB7WZUJJMAXVZHUG5H7DGWT63OOR4YX36CXHA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.108057526 (152h26m31.659527987s) [noise=true , address=77.90.31.11:4553 , piece_id=3PZ4ZNGPJW6QUF5QFZMXSUO7AGLRN3UB65SRLE26OD2EV7K7Z3KA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:34.848302803 (152h26m33.399773254s) [upload=134265]: Next returned nil segment Nov 15 16:30:34.848316729 (152h26m33.39978719s) [upload=134265]: Waiting for error group managing segments... Nov 15 16:30:35.264891445 (152h26m33.816361896s) [noise=true , address=77.90.31.19:4841 , piece_id=N6LBPIHIVTBI2S4GB7WZUJJMAXVZHUG5H7DGWT63OOR4YX36CXHA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: <nil> Nov 15 16:30:35.273189547 (152h26m33.824659998s) [noise=true , address=77.90.31.11:4553 , piece_id=3PZ4ZNGPJW6QUF5QFZMXSUO7AGLRN3UB65SRLE26OD2EV7K7Z3KA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: <nil> Nov 15 16:30:35.285617363 (152h26m33.837087814s) [noise=true , address=77.90.31.10:4519 , piece_id=KG4ZZKB4WZTVGL7XDDBGGCG57Z2WYFBXGKCTTHLFSA5VMBZ7BZVQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: <nil> Nov 15 16:30:35.353924254 (152h26m33.905394716s) [noise=true , address=45.91.116.11:4117 , piece_id=EWRHIJQXTRBQ2HJEXZ26AA23QNKNX5T5YXTLV73G7JDRU26FJGZQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: <nil> Nov 15 16:31:02.140393813 (152h27m0.691864284s) [noise=true , address=145.79.227.15:4874 , piece_id=2BVYQHWPNLMNVFY6YBPCPYCIJ4V366CPQ4ZBORU5ES6O25ZKCISA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:47072->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:47072->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47072->145.79.227.15:4874: use of closed network connection Nov 15 16:31:02.140566301 (152h27m0.692036773s) [noise=true , address=145.79.227.23:4546 , piece_id=D6U2YPSDJMP3CDRRXMHWKFHAB4MQMEDJ75F7LKDNZ5URMOIKE5UA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:43712->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:43712->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43712->145.79.227.23:4546: use of closed network connection Nov 15 16:31:02.140810807 (152h27m0.692281268s) [noise=true , address=145.79.227.27:4691 , piece_id=MISK27GKSRJKB6TN5ZHD4FWCU66XNVSUSEY5WOPFZHBN7JHNL5NQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34154->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34154->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34154->145.79.227.27:4691: use of closed network connection Nov 15 16:31:02.141099577 (152h27m0.692570038s) [noise=true , address=145.79.227.20:4437 , piece_id=CMN5H7JASFDTOSYUIPQ3UYTFJPXIF3IAY534LQLSX7C6MWFVQT7A, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:31:02.142116905 (152h27m0.693587376s) [noise=true , address=145.79.227.10:4402 , piece_id=DG4WNF766BPUCG7LTYYDMYDVGRJU66T2LQU7NHMGHZMQO24652YA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:57310->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:31:02.142622497 (152h27m0.694092978s) [noise=true , address=145.79.227.22:4509 , piece_id=6LF3P3VJEANLJCRBG6KYTMEJC4XMNQ6IPOVARRCUJWH6LSLIEYRQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:31:02.142984557 (152h27m0.694455018s) [noise=true , address=145.79.227.17:4970 , piece_id=SBE5DL43YQF5H762LPMHPWJKZTSWJDLPWDU6JQW7GROYTZZXGYLQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:44456->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:31:02.143152886 (152h27m0.694623348s) [noise=true , address=145.79.227.26:4655 , piece_id=FRJA7IRNY645QO6A4CEA34GWTIGNP7VP34IMUHSS3YGLF6XBLIOQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: read tcp 188.166.225.170:42844->145.79.227.26:4655: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:31:02.143524234 (152h27m0.694994695s) [noise=true , address=145.79.227.13:4804 , piece_id=KQ33HVOYFKXG7KWS6PQA3J37NWT2UO3S6ZOJPFRDYVTWZACLIPIA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:60332->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:60332->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60332->145.79.227.13:4804: use of closed network connection Nov 15 16:31:02.143655564 (152h27m0.695126025s) [noise=true , address=145.79.227.9:4365 , piece_id=GG6FPM3K3IGM5HBGUEISASLYQID3AI4RCRQT3LX2SRVUGYKYTWLQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:52916->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:52916->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52916->145.79.227.9:4365: use of closed network connection Nov 15 16:31:02.144053942 (152h27m0.695524403s) [noise=true , address=145.79.227.14:4840 , piece_id=KX5TTJYMQJDJVQJZAWAU4YQFYGW6OR43D5ISNKHK62ESSQE4Q4AA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:46560->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:46560->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46560->145.79.227.14:4840: use of closed network connection Nov 15 16:31:02.144548404 (152h27m0.696018865s) [noise=true , address=145.79.227.32:4836 , piece_id=VI4GQZITGKW6ONCICLUC3IYHZTJMS2PNC2N53KA6F3VQMTHA3WXQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50162->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50162->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50162->145.79.227.32:4836: use of closed network connection Nov 15 16:31:02.144589713 (152h27m0.696060174s) [seg_pos={0 0}, upload=134265]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 15 16:31:02.390629911 (152h27m0.942100372s) [upload=134265]: Done uploading segments... Nov 15 16:31:02.390680899 (152h27m0.94215136s) [upload=134265]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.077115651s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................X........X.........................] duration: 3.474452507s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:32:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.474478395s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.997639077s trace: https://jaeger.dev.storj.io/trace/6bf6128111f84872 FAIL (2025-11-15 16:31:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HK62ESSQE4Q4AA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.095222666 (152h26m31.646693117s) [noise=true , address=145.79.227.23:4546 , piece_id=D6U2YPSDJMP3CDRRXMHWKFHAB4MQMEDJ75F7LKDNZ5URMOIKE5UA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.095747606 (152h26m31.647218067s) [noise=true , address=145.79.227.10:4402 , piece_id=DG4WNF766BPUCG7LTYYDMYDVGRJU66T2LQU7NHMGHZMQO24652YA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.097728758 (152h26m31.649199219s) [noise=true , address=145.79.227.15:4874 , piece_id=2BVYQHWPNLMNVFY6YBPCPYCIJ4V366CPQ4ZBORU5ES6O25ZKCISA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.098417129 (152h26m31.64988758s) [noise=true , address=45.91.116.11:4117 , piece_id=EWRHIJQXTRBQ2HJEXZ26AA23QNKNX5T5YXTLV73G7JDRU26FJGZQ, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.100496519 (152h26m31.65196698s) [noise=true , address=145.79.227.32:4836 , piece_id=VI4GQZITGKW6ONCICLUC3IYHZTJMS2PNC2N53KA6F3VQMTHA3WXQ, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.102319971 (152h26m31.653790432s) [noise=true , address=145.79.227.9:4365 , piece_id=GG6FPM3K3IGM5HBGUEISASLYQID3AI4RCRQT3LX2SRVUGYKYTWLQ, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.103883157 (152h26m31.655353619s) [noise=true , address=145.79.227.17:4970 , piece_id=SBE5DL43YQF5H762LPMHPWJKZTSWJDLPWDU6JQW7GROYTZZXGYLQ, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.105434873 (152h26m31.656905334s) [noise=true , address=77.90.31.19:4841 , piece_id=N6LBPIHIVTBI2S4GB7WZUJJMAXVZHUG5H7DGWT63OOR4YX36CXHA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.108057526 (152h26m31.659527987s) [noise=true , address=77.90.31.11:4553 , piece_id=3PZ4ZNGPJW6QUF5QFZMXSUO7AGLRN3UB65SRLE26OD2EV7K7Z3KA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:34.848302803 (152h26m33.399773254s) [upload=134265]: Next returned nil segment Nov 15 16:30:34.848316729 (152h26m33.39978719s) [upload=134265]: Waiting for error group managing segments... Nov 15 16:30:35.264891445 (152h26m33.816361896s) [noise=true , address=77.90.31.19:4841 , piece_id=N6LBPIHIVTBI2S4GB7WZUJJMAXVZHUG5H7DGWT63OOR4YX36CXHA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: <nil> Nov 15 16:30:35.273189547 (152h26m33.824659998s) [noise=true , address=77.90.31.11:4553 , piece_id=3PZ4ZNGPJW6QUF5QFZMXSUO7AGLRN3UB65SRLE26OD2EV7K7Z3KA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: <nil> Nov 15 16:30:35.285617363 (152h26m33.837087814s) [noise=true , address=77.90.31.10:4519 , piece_id=KG4ZZKB4WZTVGL7XDDBGGCG57Z2WYFBXGKCTTHLFSA5VMBZ7BZVQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: <nil> Nov 15 16:30:35.353924254 (152h26m33.905394716s) [noise=true , address=45.91.116.11:4117 , piece_id=EWRHIJQXTRBQ2HJEXZ26AA23QNKNX5T5YXTLV73G7JDRU26FJGZQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: <nil> Nov 15 16:31:02.140393813 (152h27m0.691864284s) [noise=true , address=145.79.227.15:4874 , piece_id=2BVYQHWPNLMNVFY6YBPCPYCIJ4V366CPQ4ZBORU5ES6O25ZKCISA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:47072->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:47072->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47072->145.79.227.15:4874: use of closed network connection Nov 15 16:31:02.140566301 (152h27m0.692036773s) [noise=true , address=145.79.227.23:4546 , piece_id=D6U2YPSDJMP3CDRRXMHWKFHAB4MQMEDJ75F7LKDNZ5URMOIKE5UA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:43712->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:43712->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43712->145.79.227.23:4546: use of closed network connection Nov 15 16:31:02.140810807 (152h27m0.692281268s) [noise=true , address=145.79.227.27:4691 , piece_id=MISK27GKSRJKB6TN5ZHD4FWCU66XNVSUSEY5WOPFZHBN7JHNL5NQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34154->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34154->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34154->145.79.227.27:4691: use of closed network connection Nov 15 16:31:02.141099577 (152h27m0.692570038s) [noise=true , address=145.79.227.20:4437 , piece_id=CMN5H7JASFDTOSYUIPQ3UYTFJPXIF3IAY534LQLSX7C6MWFVQT7A, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:31:02.142116905 (152h27m0.693587376s) [noise=true , address=145.79.227.10:4402 , piece_id=DG4WNF766BPUCG7LTYYDMYDVGRJU66T2LQU7NHMGHZMQO24652YA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:57310->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:31:02.142622497 (152h27m0.694092978s) [noise=true , address=145.79.227.22:4509 , piece_id=6LF3P3VJEANLJCRBG6KYTMEJC4XMNQ6IPOVARRCUJWH6LSLIEYRQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:31:02.142984557 (152h27m0.694455018s) [noise=true , address=145.79.227.17:4970 , piece_id=SBE5DL43YQF5H762LPMHPWJKZTSWJDLPWDU6JQW7GROYTZZXGYLQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:44456->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:31:02.143152886 (152h27m0.694623348s) [noise=true , address=145.79.227.26:4655 , piece_id=FRJA7IRNY645QO6A4CEA34GWTIGNP7VP34IMUHSS3YGLF6XBLIOQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: read tcp 188.166.225.170:42844->145.79.227.26:4655: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:31:02.143524234 (152h27m0.694994695s) [noise=true , address=145.79.227.13:4804 , piece_id=KQ33HVOYFKXG7KWS6PQA3J37NWT2UO3S6ZOJPFRDYVTWZACLIPIA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:60332->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:60332->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60332->145.79.227.13:4804: use of closed network connection Nov 15 16:31:02.143655564 (152h27m0.695126025s) [noise=true , address=145.79.227.9:4365 , piece_id=GG6FPM3K3IGM5HBGUEISASLYQID3AI4RCRQT3LX2SRVUGYKYTWLQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:52916->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:52916->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52916->145.79.227.9:4365: use of closed network connection Nov 15 16:31:02.144053942 (152h27m0.695524403s) [noise=true , address=145.79.227.14:4840 , piece_id=KX5TTJYMQJDJVQJZAWAU4YQFYGW6OR43D5ISNKHK62ESSQE4Q4AA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:46560->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:46560->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46560->145.79.227.14:4840: use of closed network connection Nov 15 16:31:02.144548404 (152h27m0.696018865s) [noise=true , address=145.79.227.32:4836 , piece_id=VI4GQZITGKW6ONCICLUC3IYHZTJMS2PNC2N53KA6F3VQMTHA3WXQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50162->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50162->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50162->145.79.227.32:4836: use of closed network connection Nov 15 16:31:02.144589713 (152h27m0.696060174s) [seg_pos={0 0}, upload=134265]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 15 16:31:02.390629911 (152h27m0.942100372s) [upload=134265]: Done uploading segments... Nov 15 16:31:02.390680899 (152h27m0.94215136s) [upload=134265]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.997668705s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 15.348459395s trace: https://jaeger.dev.storj.io/trace/6bf6128111f84872 FAIL (2025-11-15 16:31:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HK62ESSQE4Q4AA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.095222666 (152h26m31.646693117s) [noise=true , address=145.79.227.23:4546 , piece_id=D6U2YPSDJMP3CDRRXMHWKFHAB4MQMEDJ75F7LKDNZ5URMOIKE5UA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.095747606 (152h26m31.647218067s) [noise=true , address=145.79.227.10:4402 , piece_id=DG4WNF766BPUCG7LTYYDMYDVGRJU66T2LQU7NHMGHZMQO24652YA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.097728758 (152h26m31.649199219s) [noise=true , address=145.79.227.15:4874 , piece_id=2BVYQHWPNLMNVFY6YBPCPYCIJ4V366CPQ4ZBORU5ES6O25ZKCISA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.098417129 (152h26m31.64988758s) [noise=true , address=45.91.116.11:4117 , piece_id=EWRHIJQXTRBQ2HJEXZ26AA23QNKNX5T5YXTLV73G7JDRU26FJGZQ, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.100496519 (152h26m31.65196698s) [noise=true , address=145.79.227.32:4836 , piece_id=VI4GQZITGKW6ONCICLUC3IYHZTJMS2PNC2N53KA6F3VQMTHA3WXQ, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.102319971 (152h26m31.653790432s) [noise=true , address=145.79.227.9:4365 , piece_id=GG6FPM3K3IGM5HBGUEISASLYQID3AI4RCRQT3LX2SRVUGYKYTWLQ, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.103883157 (152h26m31.655353619s) [noise=true , address=145.79.227.17:4970 , piece_id=SBE5DL43YQF5H762LPMHPWJKZTSWJDLPWDU6JQW7GROYTZZXGYLQ, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.105434873 (152h26m31.656905334s) [noise=true , address=77.90.31.19:4841 , piece_id=N6LBPIHIVTBI2S4GB7WZUJJMAXVZHUG5H7DGWT63OOR4YX36CXHA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.108057526 (152h26m31.659527987s) [noise=true , address=77.90.31.11:4553 , piece_id=3PZ4ZNGPJW6QUF5QFZMXSUO7AGLRN3UB65SRLE26OD2EV7K7Z3KA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:34.848302803 (152h26m33.399773254s) [upload=134265]: Next returned nil segment Nov 15 16:30:34.848316729 (152h26m33.39978719s) [upload=134265]: Waiting for error group managing segments... Nov 15 16:30:35.264891445 (152h26m33.816361896s) [noise=true , address=77.90.31.19:4841 , piece_id=N6LBPIHIVTBI2S4GB7WZUJJMAXVZHUG5H7DGWT63OOR4YX36CXHA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: <nil> Nov 15 16:30:35.273189547 (152h26m33.824659998s) [noise=true , address=77.90.31.11:4553 , piece_id=3PZ4ZNGPJW6QUF5QFZMXSUO7AGLRN3UB65SRLE26OD2EV7K7Z3KA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: <nil> Nov 15 16:30:35.285617363 (152h26m33.837087814s) [noise=true , address=77.90.31.10:4519 , piece_id=KG4ZZKB4WZTVGL7XDDBGGCG57Z2WYFBXGKCTTHLFSA5VMBZ7BZVQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: <nil> Nov 15 16:30:35.353924254 (152h26m33.905394716s) [noise=true , address=45.91.116.11:4117 , piece_id=EWRHIJQXTRBQ2HJEXZ26AA23QNKNX5T5YXTLV73G7JDRU26FJGZQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: <nil> Nov 15 16:31:02.140393813 (152h27m0.691864284s) [noise=true , address=145.79.227.15:4874 , piece_id=2BVYQHWPNLMNVFY6YBPCPYCIJ4V366CPQ4ZBORU5ES6O25ZKCISA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:47072->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:47072->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47072->145.79.227.15:4874: use of closed network connection Nov 15 16:31:02.140566301 (152h27m0.692036773s) [noise=true , address=145.79.227.23:4546 , piece_id=D6U2YPSDJMP3CDRRXMHWKFHAB4MQMEDJ75F7LKDNZ5URMOIKE5UA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:43712->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:43712->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43712->145.79.227.23:4546: use of closed network connection Nov 15 16:31:02.140810807 (152h27m0.692281268s) [noise=true , address=145.79.227.27:4691 , piece_id=MISK27GKSRJKB6TN5ZHD4FWCU66XNVSUSEY5WOPFZHBN7JHNL5NQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34154->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34154->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34154->145.79.227.27:4691: use of closed network connection Nov 15 16:31:02.141099577 (152h27m0.692570038s) [noise=true , address=145.79.227.20:4437 , piece_id=CMN5H7JASFDTOSYUIPQ3UYTFJPXIF3IAY534LQLSX7C6MWFVQT7A, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:31:02.142116905 (152h27m0.693587376s) [noise=true , address=145.79.227.10:4402 , piece_id=DG4WNF766BPUCG7LTYYDMYDVGRJU66T2LQU7NHMGHZMQO24652YA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:57310->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:31:02.142622497 (152h27m0.694092978s) [noise=true , address=145.79.227.22:4509 , piece_id=6LF3P3VJEANLJCRBG6KYTMEJC4XMNQ6IPOVARRCUJWH6LSLIEYRQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:31:02.142984557 (152h27m0.694455018s) [noise=true , address=145.79.227.17:4970 , piece_id=SBE5DL43YQF5H762LPMHPWJKZTSWJDLPWDU6JQW7GROYTZZXGYLQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:44456->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:31:02.143152886 (152h27m0.694623348s) [noise=true , address=145.79.227.26:4655 , piece_id=FRJA7IRNY645QO6A4CEA34GWTIGNP7VP34IMUHSS3YGLF6XBLIOQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: read tcp 188.166.225.170:42844->145.79.227.26:4655: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:31:02.143524234 (152h27m0.694994695s) [noise=true , address=145.79.227.13:4804 , piece_id=KQ33HVOYFKXG7KWS6PQA3J37NWT2UO3S6ZOJPFRDYVTWZACLIPIA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:60332->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:60332->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60332->145.79.227.13:4804: use of closed network connection Nov 15 16:31:02.143655564 (152h27m0.695126025s) [noise=true , address=145.79.227.9:4365 , piece_id=GG6FPM3K3IGM5HBGUEISASLYQID3AI4RCRQT3LX2SRVUGYKYTWLQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:52916->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:52916->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52916->145.79.227.9:4365: use of closed network connection Nov 15 16:31:02.144053942 (152h27m0.695524403s) [noise=true , address=145.79.227.14:4840 , piece_id=KX5TTJYMQJDJVQJZAWAU4YQFYGW6OR43D5ISNKHK62ESSQE4Q4AA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:46560->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:46560->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46560->145.79.227.14:4840: use of closed network connection Nov 15 16:31:02.144548404 (152h27m0.696018865s) [noise=true , address=145.79.227.32:4836 , piece_id=VI4GQZITGKW6ONCICLUC3IYHZTJMS2PNC2N53KA6F3VQMTHA3WXQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50162->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50162->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50162->145.79.227.32:4836: use of closed network connection Nov 15 16:31:02.144589713 (152h27m0.696060174s) [seg_pos={0 0}, upload=134265]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 15 16:31:02.390629911 (152h27m0.942100372s) [upload=134265]: Done uploading segments... Nov 15 16:31:02.390680899 (152h27m0.94215136s) [upload=134265]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 15.348481737s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................X........X........................] duration: 3.430770466s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:31:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430790365s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.252698692s trace: https://jaeger.dev.storj.io/trace/6bf6128111f84872 FAIL (2025-11-15 16:31:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== HK62ESSQE4Q4AA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.095222666 (152h26m31.646693117s) [noise=true , address=145.79.227.23:4546 , piece_id=D6U2YPSDJMP3CDRRXMHWKFHAB4MQMEDJ75F7LKDNZ5URMOIKE5UA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.095747606 (152h26m31.647218067s) [noise=true , address=145.79.227.10:4402 , piece_id=DG4WNF766BPUCG7LTYYDMYDVGRJU66T2LQU7NHMGHZMQO24652YA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.097728758 (152h26m31.649199219s) [noise=true , address=145.79.227.15:4874 , piece_id=2BVYQHWPNLMNVFY6YBPCPYCIJ4V366CPQ4ZBORU5ES6O25ZKCISA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.098417129 (152h26m31.64988758s) [noise=true , address=45.91.116.11:4117 , piece_id=EWRHIJQXTRBQ2HJEXZ26AA23QNKNX5T5YXTLV73G7JDRU26FJGZQ, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.100496519 (152h26m31.65196698s) [noise=true , address=145.79.227.32:4836 , piece_id=VI4GQZITGKW6ONCICLUC3IYHZTJMS2PNC2N53KA6F3VQMTHA3WXQ, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.102319971 (152h26m31.653790432s) [noise=true , address=145.79.227.9:4365 , piece_id=GG6FPM3K3IGM5HBGUEISASLYQID3AI4RCRQT3LX2SRVUGYKYTWLQ, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.103883157 (152h26m31.655353619s) [noise=true , address=145.79.227.17:4970 , piece_id=SBE5DL43YQF5H762LPMHPWJKZTSWJDLPWDU6JQW7GROYTZZXGYLQ, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.105434873 (152h26m31.656905334s) [noise=true , address=77.90.31.19:4841 , piece_id=N6LBPIHIVTBI2S4GB7WZUJJMAXVZHUG5H7DGWT63OOR4YX36CXHA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:33.108057526 (152h26m31.659527987s) [noise=true , address=77.90.31.11:4553 , piece_id=3PZ4ZNGPJW6QUF5QFZMXSUO7AGLRN3UB65SRLE26OD2EV7K7Z3KA, seg_pos={0 0}, upload=134265]: Uploading piece... Nov 15 16:30:34.848302803 (152h26m33.399773254s) [upload=134265]: Next returned nil segment Nov 15 16:30:34.848316729 (152h26m33.39978719s) [upload=134265]: Waiting for error group managing segments... Nov 15 16:30:35.264891445 (152h26m33.816361896s) [noise=true , address=77.90.31.19:4841 , piece_id=N6LBPIHIVTBI2S4GB7WZUJJMAXVZHUG5H7DGWT63OOR4YX36CXHA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: <nil> Nov 15 16:30:35.273189547 (152h26m33.824659998s) [noise=true , address=77.90.31.11:4553 , piece_id=3PZ4ZNGPJW6QUF5QFZMXSUO7AGLRN3UB65SRLE26OD2EV7K7Z3KA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: <nil> Nov 15 16:30:35.285617363 (152h26m33.837087814s) [noise=true , address=77.90.31.10:4519 , piece_id=KG4ZZKB4WZTVGL7XDDBGGCG57Z2WYFBXGKCTTHLFSA5VMBZ7BZVQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: <nil> Nov 15 16:30:35.353924254 (152h26m33.905394716s) [noise=true , address=45.91.116.11:4117 , piece_id=EWRHIJQXTRBQ2HJEXZ26AA23QNKNX5T5YXTLV73G7JDRU26FJGZQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: <nil> Nov 15 16:31:02.140393813 (152h27m0.691864284s) [noise=true , address=145.79.227.15:4874 , piece_id=2BVYQHWPNLMNVFY6YBPCPYCIJ4V366CPQ4ZBORU5ES6O25ZKCISA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:47072->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:47072->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47072->145.79.227.15:4874: use of closed network connection Nov 15 16:31:02.140566301 (152h27m0.692036773s) [noise=true , address=145.79.227.23:4546 , piece_id=D6U2YPSDJMP3CDRRXMHWKFHAB4MQMEDJ75F7LKDNZ5URMOIKE5UA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:43712->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:43712->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43712->145.79.227.23:4546: use of closed network connection Nov 15 16:31:02.140810807 (152h27m0.692281268s) [noise=true , address=145.79.227.27:4691 , piece_id=MISK27GKSRJKB6TN5ZHD4FWCU66XNVSUSEY5WOPFZHBN7JHNL5NQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, address:145.79.227.27:4691): protocol: write tcp 188.166.225.170:34154->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:34154->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34154->145.79.227.27:4691: use of closed network connection Nov 15 16:31:02.141099577 (152h27m0.692570038s) [noise=true , address=145.79.227.20:4437 , piece_id=CMN5H7JASFDTOSYUIPQ3UYTFJPXIF3IAY534LQLSX7C6MWFVQT7A, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:31:02.142116905 (152h27m0.693587376s) [noise=true , address=145.79.227.10:4402 , piece_id=DG4WNF766BPUCG7LTYYDMYDVGRJU66T2LQU7NHMGHZMQO24652YA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: read tcp 188.166.225.170:57310->145.79.227.10:4402: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:31:02.142622497 (152h27m0.694092978s) [noise=true , address=145.79.227.22:4509 , piece_id=6LF3P3VJEANLJCRBG6KYTMEJC4XMNQ6IPOVARRCUJWH6LSLIEYRQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 15 16:31:02.142984557 (152h27m0.694455018s) [noise=true , address=145.79.227.17:4970 , piece_id=SBE5DL43YQF5H762LPMHPWJKZTSWJDLPWDU6JQW7GROYTZZXGYLQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:44456->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:31:02.143152886 (152h27m0.694623348s) [noise=true , address=145.79.227.26:4655 , piece_id=FRJA7IRNY645QO6A4CEA34GWTIGNP7VP34IMUHSS3YGLF6XBLIOQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: read tcp 188.166.225.170:42844->145.79.227.26:4655: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 16:31:02.143524234 (152h27m0.694994695s) [noise=true , address=145.79.227.13:4804 , piece_id=KQ33HVOYFKXG7KWS6PQA3J37NWT2UO3S6ZOJPFRDYVTWZACLIPIA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:60332->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:60332->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60332->145.79.227.13:4804: use of closed network connection Nov 15 16:31:02.143655564 (152h27m0.695126025s) [noise=true , address=145.79.227.9:4365 , piece_id=GG6FPM3K3IGM5HBGUEISASLYQID3AI4RCRQT3LX2SRVUGYKYTWLQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:52916->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:52916->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52916->145.79.227.9:4365: use of closed network connection Nov 15 16:31:02.144053942 (152h27m0.695524403s) [noise=true , address=145.79.227.14:4840 , piece_id=KX5TTJYMQJDJVQJZAWAU4YQFYGW6OR43D5ISNKHK62ESSQE4Q4AA, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:46560->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:46560->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46560->145.79.227.14:4840: use of closed network connection Nov 15 16:31:02.144548404 (152h27m0.696018865s) [noise=true , address=145.79.227.32:4836 , piece_id=VI4GQZITGKW6ONCICLUC3IYHZTJMS2PNC2N53KA6F3VQMTHA3WXQ, seg_pos={0 0}, upload=134265]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:50162->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:50162->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50162->145.79.227.32:4836: use of closed network connection Nov 15 16:31:02.144589713 (152h27m0.696060174s) [seg_pos={0 0}, upload=134265]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 15 16:31:02.390629911 (152h27m0.942100372s) [upload=134265]: Done uploading segments... Nov 15 16:31:02.390680899 (152h27m0.94215136s) [upload=134265]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.252725173s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................X........X.......................] duration: 3.604913704s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:30:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.604964121s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)