Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.........X..............X.X.........X...XX......XXXX.....X.X] duration: 5.001007054s trace: https://jaeger.dev.storj.io/trace/2aaa7efe1236e11c FAIL (2025-12-28 10:57:23 UTC) orbiter warning: context deadline exceeded Duration: 5.001029797s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..........X..............X.X.........X...XX......XXXX.....X.] duration: 5.001065503s trace: https://jaeger.dev.storj.io/trace/35a8ec1d595bc494 FAIL (2025-12-28 10:56:56 UTC) orbiter warning: context deadline exceeded Duration: 5.001089791s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....X..XX.....X..X............XX...........XXX.X.X.X...X...] duration: 7.245213534s trace: https://jaeger.dev.storj.io/trace/7a355300f6241173 FAIL (2025-12-28 10:56:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: RIM3YVQL76KGL74X32UCNKRM7HHFIOUJ2BJZDVAZ6H3V5QKIMKZQ) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: HP4BRKP7YZJFTSPOUONWDLRIVBZAESSM32YD4X4BA5V4HGU425MA) context deadline exceeded ======LOGS====== 10:53:43.585989640 (547h24m43.457072685s) [upload=480002]: Starting upload Dec 28 10:53:43.586143492 (547h24m43.457226548s) [upload=480002]: Done starting upload Dec 28 10:53:43.586264864 (547h24m43.457347909s) [upload=480002]: Uploading segments... Dec 28 10:53:43.586571898 (547h24m43.457654943s) [upload=480002]: Got next segment. Inline: false Dec 28 10:53:43.586625761 (547h24m43.457708806s) [upload=480002]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:53:43.586634778 (547h24m43.457717843s) [upload=480002]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:53:43.831274423 (547h24m43.702357458s) [seg_pos={0 0}, upload=480002]: Begin upload segment... Dec 28 10:53:43.831318016 (547h24m43.702401051s) [seg_pos={0 0}, upload=480002]: Done begin upload segment. Dec 28 10:53:43.831345458 (547h24m43.702428494s) [noise=true , address=145.79.227.14:4839 , piece_id=B7RXR35OCUBA4KONQGJYAG7MGZWMN2WY7ZJA5DYE2EAAYTWOCBLQ, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.831488691 (547h24m43.702571736s) [noise=true , address=145.79.227.17:4972 , piece_id=QCLGES7PGYR4RWFFTIOG2TNCXHRMNSLW4ZW3SO37LCVZ7DEXEBOA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.831984536 (547h24m43.703067581s) [noise=true , address=77.90.31.11:4554 , piece_id=RIM3YVQL76KGL74X32UCNKRM7HHFIOUJ2BJZDVAZ6H3V5QKIMKZQ, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.832249520 (547h24m43.703332555s) [noise=true , address=145.79.227.37:4566 , piece_id=WUNE26SHHBUBRUMKS4D3KULNFI7TBTZNF7WKCAUTWKXLHETHV5QA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.832410807 (547h24m43.703493853s) [noise=true , address=45.91.116.14:4190 , piece_id=25FPEMOAC4WWMTDYGB4YKV2BLBBT6VA4BZMM3REGR3C2MOXECBZQ, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.832682605 (547h24m43.70376564s) [noise=true , address=45.91.116.4:4937 , piece_id=QMRJT4RIWXEJQ4L2YOKUXMP6DI7ICNXNTNOOMA6QJL2SGKDOITWA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.832772406 (547h24m43.703855461s) [noise=true , address=145.79.227.13:4804 , piece_id=2YS45DKJFQACE5OZYA7FRVOEAGT4R5INCJJDK5P4EH5NU7EBGFHA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.832949423 (547h24m43.704032468s) [noise=true , address=145.79.227.4:4187 , piece_id=EFBN3QHEFEOFEQPSNMOXRNC4FIP3NQQ3AXQP6LWAXB55GH63VXLA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.833273821 (547h24m43.704356866s) [noise=true , address=145.79.227.51:4410 , piece_id=RZROMTT7LL6VZVS72TMORAW4TLFY3HG36TFBDJ5WO25BOY6CQY3Q, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.833286855 (547h24m43.704369901s) [noise=true , address=77.90.31.19:4842 , piece_id=JYUC6HPMWUBLFVNQQPM4ZM7J76DHBPXCK25XGSYXDCOOX5OTNMAQ, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.833618778 (547h24m43.704701813s) [noise=true , address=45.91.116.12:4228 , piece_id=TUOEAWNBH5HWDA5XOSZHKQL3CQKUE5IOT4VY4AJOQ4T2MQAJHFHQ, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.833855849 (547h24m43.704938884s) [noise=true , address=77.90.31.15:4697 , piece_id=TDMVGRIHVTB26UL74JTKP4WS7I6HI3EEWONUAXB7WXZWEPLLDERA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.833971800 (547h24m43.705054835s) [noise=true , address=145.79.227.29:4942 , piece_id=ZIGID4KN32ZIW3S245DR4UA6ZPHPNVEZCGFHAWD3LHSYLQQ2TW5Q, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.834577784 (547h24m43.705660829s) [noise=true , address=77.90.31.12:4590 , piece_id=HP4BRKP7YZJFTSPOUONWDLRIVBZAESSM32YD4X4BA5V4HGU425MA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.835193506 (547h24m43.706276551s) [noise=true , address=145.79.227.7:4293 , piece_id=Z53W5MKI44TL3BC4KF4TEWBESJEBEPW25IAHAPOVKI3ZS6POVZAA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.835768911 (547h24m43.706851957s) [noise=true , address=45.91.116.5:4297 , piece_id=YO7ASJXSJABEQSQWLVWAID4TLFLS5J5FWCEG5DLFRFTJMH6JIJ4A, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:45.561353437 (547h24m45.432436482s) [upload=480002]: Next returned nil segment Dec 28 10:53:45.561386249 (547h24m45.432469284s) [upload=480002]: Waiting for error group managing segments... Dec 28 10:53:45.839062402 (547h24m45.710145437s) [noise=true , address=145.79.227.14:4839 , piece_id=B7RXR35OCUBA4KONQGJYAG7MGZWMN2WY7ZJA5DYE2EAAYTWOCBLQ, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.847278817 (547h24m45.718361852s) [noise=true , address=145.79.227.37:4566 , piece_id=WUNE26SHHBUBRUMKS4D3KULNFI7TBTZNF7WKCAUTWKXLHETHV5QA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.847713345 (547h24m45.71879638s) [noise=true , address=145.79.227.13:4804 , piece_id=2YS45DKJFQACE5OZYA7FRVOEAGT4R5INCJJDK5P4EH5NU7EBGFHA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.848390675 (547h24m45.71947371s) [noise=true , address=145.79.227.4:4187 , piece_id=EFBN3QHEFEOFEQPSNMOXRNC4FIP3NQQ3AXQP6LWAXB55GH63VXLA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.849158426 (547h24m45.720241472s) [noise=true , address=145.79.227.51:4410 , piece_id=RZROMTT7LL6VZVS72TMORAW4TLFY3HG36TFBDJ5WO25BOY6CQY3Q, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.861271051 (547h24m45.732354096s) [noise=true , address=145.79.227.17:4972 , piece_id=QCLGES7PGYR4RWFFTIOG2TNCXHRMNSLW4ZW3SO37LCVZ7DEXEBOA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.865562754 (547h24m45.736645789s) [noise=true , address=145.79.227.29:4942 , piece_id=ZIGID4KN32ZIW3S245DR4UA6ZPHPNVEZCGFHAWD3LHSYLQQ2TW5Q, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.882953528 (547h24m45.754036563s) [noise=true , address=145.79.227.7:4293 , piece_id=Z53W5MKI44TL3BC4KF4TEWBESJEBEPW25IAHAPOVKI3ZS6POVZAA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.012631097 (547h24m45.883714132s) [noise=true , address=77.90.31.12:4590 , piece_id=HP4BRKP7YZJFTSPOUONWDLRIVBZAESSM32YD4X4BA5V4HGU425MA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.067077802 (547h24m45.938160838s) [noise=true , address=45.91.116.5:4297 , piece_id=YO7ASJXSJABEQSQWLVWAID4TLFLS5J5FWCEG5DLFRFTJMH6JIJ4A, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.068479281 (547h24m45.939562316s) [noise=true , address=45.91.116.4:4937 , piece_id=QMRJT4RIWXEJQ4L2YOKUXMP6DI7ICNXNTNOOMA6QJL2SGKDOITWA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.092376034 (547h24m45.963459069s) [noise=true , address=77.90.31.11:4554 , piece_id=RIM3YVQL76KGL74X32UCNKRM7HHFIOUJ2BJZDVAZ6H3V5QKIMKZQ, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.155681188 (547h24m46.026764233s) [noise=true , address=77.90.31.19:4842 , piece_id=JYUC6HPMWUBLFVNQQPM4ZM7J76DHBPXCK25XGSYXDCOOX5OTNMAQ, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.155707277 (547h24m46.026790312s) [seg_pos={0 0}, upload=480002]: Segment reached optimal threshold of 13 pieces. Dec 28 10:53:46.155967233 (547h24m46.027050288s) [noise=true , address=77.90.31.15:4697 , piece_id=TDMVGRIHVTB26UL74JTKP4WS7I6HI3EEWONUAXB7WXZWEPLLDERA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 10:53:46.155976019 (547h24m46.027059064s) [noise=true , address=77.90.31.15:4697 , piece_id=TDMVGRIHVTB26UL74JTKP4WS7I6HI3EEWONUAXB7WXZWEPLLDERA, seg_pos={0 0}, upload=480002]: Long tail cancellation due to optimal threshold Dec 28 10:53:46.156065570 (547h24m46.027148625s) [noise=true , address=45.91.116.14:4190 , piece_id=25FPEMOAC4WWMTDYGB4YKV2BLBBT6VA4BZMM3REGR3C2MOXECBZQ, seg_pos={0 0}, upload=480002]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:54120->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:54120->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54120->45.91.116.14:4190: use of closed network connection Dec 28 10:53:46.156076060 (547h24m46.027159105s) [noise=true , address=45.91.116.14:4190 , piece_id=25FPEMOAC4WWMTDYGB4YKV2BLBBT6VA4BZMM3REGR3C2MOXECBZQ, seg_pos={0 0}, upload=480002]: Long tail cancellation due to optimal threshold Dec 28 10:53:46.156341756 (547h24m46.027424791s) [noise=true , address=45.91.116.12:4228 , piece_id=TUOEAWNBH5HWDA5XOSZHKQL3CQKUE5IOT4VY4AJOQ4T2MQAJHFHQ, seg_pos={0 0}, upload=480002]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 10:53:46.156351585 (547h24m46.02743463s) [noise=true , address=45.91.116.12:4228 , piece_id=TUOEAWNBH5HWDA5XOSZHKQL3CQKUE5IOT4VY4AJOQ4T2MQAJHFHQ, seg_pos={0 0}, upload=480002]: Long tail cancellation due to optimal threshold Dec 28 10:53:46.156379498 (547h24m46.027462533s) [seg_pos={0 0}, upload=480002]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:53:46.156485770 (547h24m46.027568805s) [upload=480002]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:53:46.156492503 (547h24m46.027575548s) [upload=480002]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:53:46.441783576 (547h24m46.312866621s) [upload=480002]: Done uploading segments... Dec 28 10:53:46.441827880 (547h24m46.312910915s) [upload=480002]: Upload finished. err: <nil> Duration: 7.245255473s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [...........X..............X.X.........X...XX......XXXX.....X] duration: 5.000538401s trace: https://jaeger.dev.storj.io/trace/35a8ec1d595bc494 FAIL (2025-12-28 10:56:19 UTC) orbiter warning: context deadline exceeded Duration: 5.000560413s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [......X..XX.....X..X............XX...........XXX.X.X.X...X..] duration: 8.410220672s trace: https://jaeger.dev.storj.io/trace/7a355300f6241173 FAIL (2025-12-28 10:55:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: RIM3YVQL76KGL74X32UCNKRM7HHFIOUJ2BJZDVAZ6H3V5QKIMKZQ) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: HP4BRKP7YZJFTSPOUONWDLRIVBZAESSM32YD4X4BA5V4HGU425MA) context deadline exceeded ======LOGS====== 10:53:43.585989640 (547h24m43.457072685s) [upload=480002]: Starting upload Dec 28 10:53:43.586143492 (547h24m43.457226548s) [upload=480002]: Done starting upload Dec 28 10:53:43.586264864 (547h24m43.457347909s) [upload=480002]: Uploading segments... Dec 28 10:53:43.586571898 (547h24m43.457654943s) [upload=480002]: Got next segment. Inline: false Dec 28 10:53:43.586625761 (547h24m43.457708806s) [upload=480002]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:53:43.586634778 (547h24m43.457717843s) [upload=480002]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:53:43.831274423 (547h24m43.702357458s) [seg_pos={0 0}, upload=480002]: Begin upload segment... Dec 28 10:53:43.831318016 (547h24m43.702401051s) [seg_pos={0 0}, upload=480002]: Done begin upload segment. Dec 28 10:53:43.831345458 (547h24m43.702428494s) [noise=true , address=145.79.227.14:4839 , piece_id=B7RXR35OCUBA4KONQGJYAG7MGZWMN2WY7ZJA5DYE2EAAYTWOCBLQ, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.831488691 (547h24m43.702571736s) [noise=true , address=145.79.227.17:4972 , piece_id=QCLGES7PGYR4RWFFTIOG2TNCXHRMNSLW4ZW3SO37LCVZ7DEXEBOA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.831984536 (547h24m43.703067581s) [noise=true , address=77.90.31.11:4554 , piece_id=RIM3YVQL76KGL74X32UCNKRM7HHFIOUJ2BJZDVAZ6H3V5QKIMKZQ, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.832249520 (547h24m43.703332555s) [noise=true , address=145.79.227.37:4566 , piece_id=WUNE26SHHBUBRUMKS4D3KULNFI7TBTZNF7WKCAUTWKXLHETHV5QA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.832410807 (547h24m43.703493853s) [noise=true , address=45.91.116.14:4190 , piece_id=25FPEMOAC4WWMTDYGB4YKV2BLBBT6VA4BZMM3REGR3C2MOXECBZQ, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.832682605 (547h24m43.70376564s) [noise=true , address=45.91.116.4:4937 , piece_id=QMRJT4RIWXEJQ4L2YOKUXMP6DI7ICNXNTNOOMA6QJL2SGKDOITWA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.832772406 (547h24m43.703855461s) [noise=true , address=145.79.227.13:4804 , piece_id=2YS45DKJFQACE5OZYA7FRVOEAGT4R5INCJJDK5P4EH5NU7EBGFHA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.832949423 (547h24m43.704032468s) [noise=true , address=145.79.227.4:4187 , piece_id=EFBN3QHEFEOFEQPSNMOXRNC4FIP3NQQ3AXQP6LWAXB55GH63VXLA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.833273821 (547h24m43.704356866s) [noise=true , address=145.79.227.51:4410 , piece_id=RZROMTT7LL6VZVS72TMORAW4TLFY3HG36TFBDJ5WO25BOY6CQY3Q, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.833286855 (547h24m43.704369901s) [noise=true , address=77.90.31.19:4842 , piece_id=JYUC6HPMWUBLFVNQQPM4ZM7J76DHBPXCK25XGSYXDCOOX5OTNMAQ, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.833618778 (547h24m43.704701813s) [noise=true , address=45.91.116.12:4228 , piece_id=TUOEAWNBH5HWDA5XOSZHKQL3CQKUE5IOT4VY4AJOQ4T2MQAJHFHQ, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.833855849 (547h24m43.704938884s) [noise=true , address=77.90.31.15:4697 , piece_id=TDMVGRIHVTB26UL74JTKP4WS7I6HI3EEWONUAXB7WXZWEPLLDERA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.833971800 (547h24m43.705054835s) [noise=true , address=145.79.227.29:4942 , piece_id=ZIGID4KN32ZIW3S245DR4UA6ZPHPNVEZCGFHAWD3LHSYLQQ2TW5Q, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.834577784 (547h24m43.705660829s) [noise=true , address=77.90.31.12:4590 , piece_id=HP4BRKP7YZJFTSPOUONWDLRIVBZAESSM32YD4X4BA5V4HGU425MA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.835193506 (547h24m43.706276551s) [noise=true , address=145.79.227.7:4293 , piece_id=Z53W5MKI44TL3BC4KF4TEWBESJEBEPW25IAHAPOVKI3ZS6POVZAA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.835768911 (547h24m43.706851957s) [noise=true , address=45.91.116.5:4297 , piece_id=YO7ASJXSJABEQSQWLVWAID4TLFLS5J5FWCEG5DLFRFTJMH6JIJ4A, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:45.561353437 (547h24m45.432436482s) [upload=480002]: Next returned nil segment Dec 28 10:53:45.561386249 (547h24m45.432469284s) [upload=480002]: Waiting for error group managing segments... Dec 28 10:53:45.839062402 (547h24m45.710145437s) [noise=true , address=145.79.227.14:4839 , piece_id=B7RXR35OCUBA4KONQGJYAG7MGZWMN2WY7ZJA5DYE2EAAYTWOCBLQ, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.847278817 (547h24m45.718361852s) [noise=true , address=145.79.227.37:4566 , piece_id=WUNE26SHHBUBRUMKS4D3KULNFI7TBTZNF7WKCAUTWKXLHETHV5QA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.847713345 (547h24m45.71879638s) [noise=true , address=145.79.227.13:4804 , piece_id=2YS45DKJFQACE5OZYA7FRVOEAGT4R5INCJJDK5P4EH5NU7EBGFHA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.848390675 (547h24m45.71947371s) [noise=true , address=145.79.227.4:4187 , piece_id=EFBN3QHEFEOFEQPSNMOXRNC4FIP3NQQ3AXQP6LWAXB55GH63VXLA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.849158426 (547h24m45.720241472s) [noise=true , address=145.79.227.51:4410 , piece_id=RZROMTT7LL6VZVS72TMORAW4TLFY3HG36TFBDJ5WO25BOY6CQY3Q, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.861271051 (547h24m45.732354096s) [noise=true , address=145.79.227.17:4972 , piece_id=QCLGES7PGYR4RWFFTIOG2TNCXHRMNSLW4ZW3SO37LCVZ7DEXEBOA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.865562754 (547h24m45.736645789s) [noise=true , address=145.79.227.29:4942 , piece_id=ZIGID4KN32ZIW3S245DR4UA6ZPHPNVEZCGFHAWD3LHSYLQQ2TW5Q, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.882953528 (547h24m45.754036563s) [noise=true , address=145.79.227.7:4293 , piece_id=Z53W5MKI44TL3BC4KF4TEWBESJEBEPW25IAHAPOVKI3ZS6POVZAA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.012631097 (547h24m45.883714132s) [noise=true , address=77.90.31.12:4590 , piece_id=HP4BRKP7YZJFTSPOUONWDLRIVBZAESSM32YD4X4BA5V4HGU425MA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.067077802 (547h24m45.938160838s) [noise=true , address=45.91.116.5:4297 , piece_id=YO7ASJXSJABEQSQWLVWAID4TLFLS5J5FWCEG5DLFRFTJMH6JIJ4A, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.068479281 (547h24m45.939562316s) [noise=true , address=45.91.116.4:4937 , piece_id=QMRJT4RIWXEJQ4L2YOKUXMP6DI7ICNXNTNOOMA6QJL2SGKDOITWA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.092376034 (547h24m45.963459069s) [noise=true , address=77.90.31.11:4554 , piece_id=RIM3YVQL76KGL74X32UCNKRM7HHFIOUJ2BJZDVAZ6H3V5QKIMKZQ, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.155681188 (547h24m46.026764233s) [noise=true , address=77.90.31.19:4842 , piece_id=JYUC6HPMWUBLFVNQQPM4ZM7J76DHBPXCK25XGSYXDCOOX5OTNMAQ, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.155707277 (547h24m46.026790312s) [seg_pos={0 0}, upload=480002]: Segment reached optimal threshold of 13 pieces. Dec 28 10:53:46.155967233 (547h24m46.027050288s) [noise=true , address=77.90.31.15:4697 , piece_id=TDMVGRIHVTB26UL74JTKP4WS7I6HI3EEWONUAXB7WXZWEPLLDERA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 10:53:46.155976019 (547h24m46.027059064s) [noise=true , address=77.90.31.15:4697 , piece_id=TDMVGRIHVTB26UL74JTKP4WS7I6HI3EEWONUAXB7WXZWEPLLDERA, seg_pos={0 0}, upload=480002]: Long tail cancellation due to optimal threshold Dec 28 10:53:46.156065570 (547h24m46.027148625s) [noise=true , address=45.91.116.14:4190 , piece_id=25FPEMOAC4WWMTDYGB4YKV2BLBBT6VA4BZMM3REGR3C2MOXECBZQ, seg_pos={0 0}, upload=480002]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:54120->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:54120->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54120->45.91.116.14:4190: use of closed network connection Dec 28 10:53:46.156076060 (547h24m46.027159105s) [noise=true , address=45.91.116.14:4190 , piece_id=25FPEMOAC4WWMTDYGB4YKV2BLBBT6VA4BZMM3REGR3C2MOXECBZQ, seg_pos={0 0}, upload=480002]: Long tail cancellation due to optimal threshold Dec 28 10:53:46.156341756 (547h24m46.027424791s) [noise=true , address=45.91.116.12:4228 , piece_id=TUOEAWNBH5HWDA5XOSZHKQL3CQKUE5IOT4VY4AJOQ4T2MQAJHFHQ, seg_pos={0 0}, upload=480002]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 10:53:46.156351585 (547h24m46.02743463s) [noise=true , address=45.91.116.12:4228 , piece_id=TUOEAWNBH5HWDA5XOSZHKQL3CQKUE5IOT4VY4AJOQ4T2MQAJHFHQ, seg_pos={0 0}, upload=480002]: Long tail cancellation due to optimal threshold Dec 28 10:53:46.156379498 (547h24m46.027462533s) [seg_pos={0 0}, upload=480002]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:53:46.156485770 (547h24m46.027568805s) [upload=480002]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:53:46.156492503 (547h24m46.027575548s) [upload=480002]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:53:46.441783576 (547h24m46.312866621s) [upload=480002]: Done uploading segments... Dec 28 10:53:46.441827880 (547h24m46.312910915s) [upload=480002]: Upload finished. err: <nil> Duration: 8.410248547s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [............X..............X.X.........X...XX......XXXX.....] duration: 3.475838375s trace: https://jaeger.dev.storj.io/trace/286db894f15472df FAIL (2025-12-28 10:55:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.475863804s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......X..XX.....X..X............XX...........XXX.X.X.X...X.] duration: 5.943005498s trace: https://jaeger.dev.storj.io/trace/7a355300f6241173 FAIL (2025-12-28 10:54:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: RIM3YVQL76KGL74X32UCNKRM7HHFIOUJ2BJZDVAZ6H3V5QKIMKZQ) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: HP4BRKP7YZJFTSPOUONWDLRIVBZAESSM32YD4X4BA5V4HGU425MA) context deadline exceeded ======LOGS====== 10:53:43.585989640 (547h24m43.457072685s) [upload=480002]: Starting upload Dec 28 10:53:43.586143492 (547h24m43.457226548s) [upload=480002]: Done starting upload Dec 28 10:53:43.586264864 (547h24m43.457347909s) [upload=480002]: Uploading segments... Dec 28 10:53:43.586571898 (547h24m43.457654943s) [upload=480002]: Got next segment. Inline: false Dec 28 10:53:43.586625761 (547h24m43.457708806s) [upload=480002]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:53:43.586634778 (547h24m43.457717843s) [upload=480002]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:53:43.831274423 (547h24m43.702357458s) [seg_pos={0 0}, upload=480002]: Begin upload segment... Dec 28 10:53:43.831318016 (547h24m43.702401051s) [seg_pos={0 0}, upload=480002]: Done begin upload segment. Dec 28 10:53:43.831345458 (547h24m43.702428494s) [noise=true , address=145.79.227.14:4839 , piece_id=B7RXR35OCUBA4KONQGJYAG7MGZWMN2WY7ZJA5DYE2EAAYTWOCBLQ, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.831488691 (547h24m43.702571736s) [noise=true , address=145.79.227.17:4972 , piece_id=QCLGES7PGYR4RWFFTIOG2TNCXHRMNSLW4ZW3SO37LCVZ7DEXEBOA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.831984536 (547h24m43.703067581s) [noise=true , address=77.90.31.11:4554 , piece_id=RIM3YVQL76KGL74X32UCNKRM7HHFIOUJ2BJZDVAZ6H3V5QKIMKZQ, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.832249520 (547h24m43.703332555s) [noise=true , address=145.79.227.37:4566 , piece_id=WUNE26SHHBUBRUMKS4D3KULNFI7TBTZNF7WKCAUTWKXLHETHV5QA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.832410807 (547h24m43.703493853s) [noise=true , address=45.91.116.14:4190 , piece_id=25FPEMOAC4WWMTDYGB4YKV2BLBBT6VA4BZMM3REGR3C2MOXECBZQ, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.832682605 (547h24m43.70376564s) [noise=true , address=45.91.116.4:4937 , piece_id=QMRJT4RIWXEJQ4L2YOKUXMP6DI7ICNXNTNOOMA6QJL2SGKDOITWA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.832772406 (547h24m43.703855461s) [noise=true , address=145.79.227.13:4804 , piece_id=2YS45DKJFQACE5OZYA7FRVOEAGT4R5INCJJDK5P4EH5NU7EBGFHA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.832949423 (547h24m43.704032468s) [noise=true , address=145.79.227.4:4187 , piece_id=EFBN3QHEFEOFEQPSNMOXRNC4FIP3NQQ3AXQP6LWAXB55GH63VXLA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.833273821 (547h24m43.704356866s) [noise=true , address=145.79.227.51:4410 , piece_id=RZROMTT7LL6VZVS72TMORAW4TLFY3HG36TFBDJ5WO25BOY6CQY3Q, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.833286855 (547h24m43.704369901s) [noise=true , address=77.90.31.19:4842 , piece_id=JYUC6HPMWUBLFVNQQPM4ZM7J76DHBPXCK25XGSYXDCOOX5OTNMAQ, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.833618778 (547h24m43.704701813s) [noise=true , address=45.91.116.12:4228 , piece_id=TUOEAWNBH5HWDA5XOSZHKQL3CQKUE5IOT4VY4AJOQ4T2MQAJHFHQ, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.833855849 (547h24m43.704938884s) [noise=true , address=77.90.31.15:4697 , piece_id=TDMVGRIHVTB26UL74JTKP4WS7I6HI3EEWONUAXB7WXZWEPLLDERA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.833971800 (547h24m43.705054835s) [noise=true , address=145.79.227.29:4942 , piece_id=ZIGID4KN32ZIW3S245DR4UA6ZPHPNVEZCGFHAWD3LHSYLQQ2TW5Q, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.834577784 (547h24m43.705660829s) [noise=true , address=77.90.31.12:4590 , piece_id=HP4BRKP7YZJFTSPOUONWDLRIVBZAESSM32YD4X4BA5V4HGU425MA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.835193506 (547h24m43.706276551s) [noise=true , address=145.79.227.7:4293 , piece_id=Z53W5MKI44TL3BC4KF4TEWBESJEBEPW25IAHAPOVKI3ZS6POVZAA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.835768911 (547h24m43.706851957s) [noise=true , address=45.91.116.5:4297 , piece_id=YO7ASJXSJABEQSQWLVWAID4TLFLS5J5FWCEG5DLFRFTJMH6JIJ4A, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:45.561353437 (547h24m45.432436482s) [upload=480002]: Next returned nil segment Dec 28 10:53:45.561386249 (547h24m45.432469284s) [upload=480002]: Waiting for error group managing segments... Dec 28 10:53:45.839062402 (547h24m45.710145437s) [noise=true , address=145.79.227.14:4839 , piece_id=B7RXR35OCUBA4KONQGJYAG7MGZWMN2WY7ZJA5DYE2EAAYTWOCBLQ, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.847278817 (547h24m45.718361852s) [noise=true , address=145.79.227.37:4566 , piece_id=WUNE26SHHBUBRUMKS4D3KULNFI7TBTZNF7WKCAUTWKXLHETHV5QA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.847713345 (547h24m45.71879638s) [noise=true , address=145.79.227.13:4804 , piece_id=2YS45DKJFQACE5OZYA7FRVOEAGT4R5INCJJDK5P4EH5NU7EBGFHA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.848390675 (547h24m45.71947371s) [noise=true , address=145.79.227.4:4187 , piece_id=EFBN3QHEFEOFEQPSNMOXRNC4FIP3NQQ3AXQP6LWAXB55GH63VXLA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.849158426 (547h24m45.720241472s) [noise=true , address=145.79.227.51:4410 , piece_id=RZROMTT7LL6VZVS72TMORAW4TLFY3HG36TFBDJ5WO25BOY6CQY3Q, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.861271051 (547h24m45.732354096s) [noise=true , address=145.79.227.17:4972 , piece_id=QCLGES7PGYR4RWFFTIOG2TNCXHRMNSLW4ZW3SO37LCVZ7DEXEBOA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.865562754 (547h24m45.736645789s) [noise=true , address=145.79.227.29:4942 , piece_id=ZIGID4KN32ZIW3S245DR4UA6ZPHPNVEZCGFHAWD3LHSYLQQ2TW5Q, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.882953528 (547h24m45.754036563s) [noise=true , address=145.79.227.7:4293 , piece_id=Z53W5MKI44TL3BC4KF4TEWBESJEBEPW25IAHAPOVKI3ZS6POVZAA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.012631097 (547h24m45.883714132s) [noise=true , address=77.90.31.12:4590 , piece_id=HP4BRKP7YZJFTSPOUONWDLRIVBZAESSM32YD4X4BA5V4HGU425MA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.067077802 (547h24m45.938160838s) [noise=true , address=45.91.116.5:4297 , piece_id=YO7ASJXSJABEQSQWLVWAID4TLFLS5J5FWCEG5DLFRFTJMH6JIJ4A, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.068479281 (547h24m45.939562316s) [noise=true , address=45.91.116.4:4937 , piece_id=QMRJT4RIWXEJQ4L2YOKUXMP6DI7ICNXNTNOOMA6QJL2SGKDOITWA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.092376034 (547h24m45.963459069s) [noise=true , address=77.90.31.11:4554 , piece_id=RIM3YVQL76KGL74X32UCNKRM7HHFIOUJ2BJZDVAZ6H3V5QKIMKZQ, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.155681188 (547h24m46.026764233s) [noise=true , address=77.90.31.19:4842 , piece_id=JYUC6HPMWUBLFVNQQPM4ZM7J76DHBPXCK25XGSYXDCOOX5OTNMAQ, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.155707277 (547h24m46.026790312s) [seg_pos={0 0}, upload=480002]: Segment reached optimal threshold of 13 pieces. Dec 28 10:53:46.155967233 (547h24m46.027050288s) [noise=true , address=77.90.31.15:4697 , piece_id=TDMVGRIHVTB26UL74JTKP4WS7I6HI3EEWONUAXB7WXZWEPLLDERA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 10:53:46.155976019 (547h24m46.027059064s) [noise=true , address=77.90.31.15:4697 , piece_id=TDMVGRIHVTB26UL74JTKP4WS7I6HI3EEWONUAXB7WXZWEPLLDERA, seg_pos={0 0}, upload=480002]: Long tail cancellation due to optimal threshold Dec 28 10:53:46.156065570 (547h24m46.027148625s) [noise=true , address=45.91.116.14:4190 , piece_id=25FPEMOAC4WWMTDYGB4YKV2BLBBT6VA4BZMM3REGR3C2MOXECBZQ, seg_pos={0 0}, upload=480002]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:54120->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:54120->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54120->45.91.116.14:4190: use of closed network connection Dec 28 10:53:46.156076060 (547h24m46.027159105s) [noise=true , address=45.91.116.14:4190 , piece_id=25FPEMOAC4WWMTDYGB4YKV2BLBBT6VA4BZMM3REGR3C2MOXECBZQ, seg_pos={0 0}, upload=480002]: Long tail cancellation due to optimal threshold Dec 28 10:53:46.156341756 (547h24m46.027424791s) [noise=true , address=45.91.116.12:4228 , piece_id=TUOEAWNBH5HWDA5XOSZHKQL3CQKUE5IOT4VY4AJOQ4T2MQAJHFHQ, seg_pos={0 0}, upload=480002]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 10:53:46.156351585 (547h24m46.02743463s) [noise=true , address=45.91.116.12:4228 , piece_id=TUOEAWNBH5HWDA5XOSZHKQL3CQKUE5IOT4VY4AJOQ4T2MQAJHFHQ, seg_pos={0 0}, upload=480002]: Long tail cancellation due to optimal threshold Dec 28 10:53:46.156379498 (547h24m46.027462533s) [seg_pos={0 0}, upload=480002]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:53:46.156485770 (547h24m46.027568805s) [upload=480002]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:53:46.156492503 (547h24m46.027575548s) [upload=480002]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:53:46.441783576 (547h24m46.312866621s) [upload=480002]: Done uploading segments... Dec 28 10:53:46.441827880 (547h24m46.312910915s) [upload=480002]: Upload finished. err: <nil> Duration: 5.943032559s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.............X..............X.X.........X...XX......XXXX....] duration: 3.399087146s trace: https://jaeger.dev.storj.io/trace/286db894f15472df FAIL (2025-12-28 10:54:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.399117034s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [........X..XX.....X..X............XX...........XXX.X.X.X...X] duration: 30.002909757s trace: https://jaeger.dev.storj.io/trace/7a355300f6241173 FAIL (2025-12-28 10:54:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: RIM3YVQL76KGL74X32UCNKRM7HHFIOUJ2BJZDVAZ6H3V5QKIMKZQ) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: HP4BRKP7YZJFTSPOUONWDLRIVBZAESSM32YD4X4BA5V4HGU425MA) context deadline exceeded ======LOGS====== 10:53:43.585989640 (547h24m43.457072685s) [upload=480002]: Starting upload Dec 28 10:53:43.586143492 (547h24m43.457226548s) [upload=480002]: Done starting upload Dec 28 10:53:43.586264864 (547h24m43.457347909s) [upload=480002]: Uploading segments... Dec 28 10:53:43.586571898 (547h24m43.457654943s) [upload=480002]: Got next segment. Inline: false Dec 28 10:53:43.586625761 (547h24m43.457708806s) [upload=480002]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:53:43.586634778 (547h24m43.457717843s) [upload=480002]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:53:43.831274423 (547h24m43.702357458s) [seg_pos={0 0}, upload=480002]: Begin upload segment... Dec 28 10:53:43.831318016 (547h24m43.702401051s) [seg_pos={0 0}, upload=480002]: Done begin upload segment. Dec 28 10:53:43.831345458 (547h24m43.702428494s) [noise=true , address=145.79.227.14:4839 , piece_id=B7RXR35OCUBA4KONQGJYAG7MGZWMN2WY7ZJA5DYE2EAAYTWOCBLQ, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.831488691 (547h24m43.702571736s) [noise=true , address=145.79.227.17:4972 , piece_id=QCLGES7PGYR4RWFFTIOG2TNCXHRMNSLW4ZW3SO37LCVZ7DEXEBOA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.831984536 (547h24m43.703067581s) [noise=true , address=77.90.31.11:4554 , piece_id=RIM3YVQL76KGL74X32UCNKRM7HHFIOUJ2BJZDVAZ6H3V5QKIMKZQ, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.832249520 (547h24m43.703332555s) [noise=true , address=145.79.227.37:4566 , piece_id=WUNE26SHHBUBRUMKS4D3KULNFI7TBTZNF7WKCAUTWKXLHETHV5QA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.832410807 (547h24m43.703493853s) [noise=true , address=45.91.116.14:4190 , piece_id=25FPEMOAC4WWMTDYGB4YKV2BLBBT6VA4BZMM3REGR3C2MOXECBZQ, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.832682605 (547h24m43.70376564s) [noise=true , address=45.91.116.4:4937 , piece_id=QMRJT4RIWXEJQ4L2YOKUXMP6DI7ICNXNTNOOMA6QJL2SGKDOITWA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.832772406 (547h24m43.703855461s) [noise=true , address=145.79.227.13:4804 , piece_id=2YS45DKJFQACE5OZYA7FRVOEAGT4R5INCJJDK5P4EH5NU7EBGFHA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.832949423 (547h24m43.704032468s) [noise=true , address=145.79.227.4:4187 , piece_id=EFBN3QHEFEOFEQPSNMOXRNC4FIP3NQQ3AXQP6LWAXB55GH63VXLA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.833273821 (547h24m43.704356866s) [noise=true , address=145.79.227.51:4410 , piece_id=RZROMTT7LL6VZVS72TMORAW4TLFY3HG36TFBDJ5WO25BOY6CQY3Q, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.833286855 (547h24m43.704369901s) [noise=true , address=77.90.31.19:4842 , piece_id=JYUC6HPMWUBLFVNQQPM4ZM7J76DHBPXCK25XGSYXDCOOX5OTNMAQ, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.833618778 (547h24m43.704701813s) [noise=true , address=45.91.116.12:4228 , piece_id=TUOEAWNBH5HWDA5XOSZHKQL3CQKUE5IOT4VY4AJOQ4T2MQAJHFHQ, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.833855849 (547h24m43.704938884s) [noise=true , address=77.90.31.15:4697 , piece_id=TDMVGRIHVTB26UL74JTKP4WS7I6HI3EEWONUAXB7WXZWEPLLDERA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.833971800 (547h24m43.705054835s) [noise=true , address=145.79.227.29:4942 , piece_id=ZIGID4KN32ZIW3S245DR4UA6ZPHPNVEZCGFHAWD3LHSYLQQ2TW5Q, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.834577784 (547h24m43.705660829s) [noise=true , address=77.90.31.12:4590 , piece_id=HP4BRKP7YZJFTSPOUONWDLRIVBZAESSM32YD4X4BA5V4HGU425MA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.835193506 (547h24m43.706276551s) [noise=true , address=145.79.227.7:4293 , piece_id=Z53W5MKI44TL3BC4KF4TEWBESJEBEPW25IAHAPOVKI3ZS6POVZAA, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:43.835768911 (547h24m43.706851957s) [noise=true , address=45.91.116.5:4297 , piece_id=YO7ASJXSJABEQSQWLVWAID4TLFLS5J5FWCEG5DLFRFTJMH6JIJ4A, seg_pos={0 0}, upload=480002]: Uploading piece... Dec 28 10:53:45.561353437 (547h24m45.432436482s) [upload=480002]: Next returned nil segment Dec 28 10:53:45.561386249 (547h24m45.432469284s) [upload=480002]: Waiting for error group managing segments... Dec 28 10:53:45.839062402 (547h24m45.710145437s) [noise=true , address=145.79.227.14:4839 , piece_id=B7RXR35OCUBA4KONQGJYAG7MGZWMN2WY7ZJA5DYE2EAAYTWOCBLQ, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.847278817 (547h24m45.718361852s) [noise=true , address=145.79.227.37:4566 , piece_id=WUNE26SHHBUBRUMKS4D3KULNFI7TBTZNF7WKCAUTWKXLHETHV5QA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.847713345 (547h24m45.71879638s) [noise=true , address=145.79.227.13:4804 , piece_id=2YS45DKJFQACE5OZYA7FRVOEAGT4R5INCJJDK5P4EH5NU7EBGFHA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.848390675 (547h24m45.71947371s) [noise=true , address=145.79.227.4:4187 , piece_id=EFBN3QHEFEOFEQPSNMOXRNC4FIP3NQQ3AXQP6LWAXB55GH63VXLA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.849158426 (547h24m45.720241472s) [noise=true , address=145.79.227.51:4410 , piece_id=RZROMTT7LL6VZVS72TMORAW4TLFY3HG36TFBDJ5WO25BOY6CQY3Q, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.861271051 (547h24m45.732354096s) [noise=true , address=145.79.227.17:4972 , piece_id=QCLGES7PGYR4RWFFTIOG2TNCXHRMNSLW4ZW3SO37LCVZ7DEXEBOA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.865562754 (547h24m45.736645789s) [noise=true , address=145.79.227.29:4942 , piece_id=ZIGID4KN32ZIW3S245DR4UA6ZPHPNVEZCGFHAWD3LHSYLQQ2TW5Q, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:45.882953528 (547h24m45.754036563s) [noise=true , address=145.79.227.7:4293 , piece_id=Z53W5MKI44TL3BC4KF4TEWBESJEBEPW25IAHAPOVKI3ZS6POVZAA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.012631097 (547h24m45.883714132s) [noise=true , address=77.90.31.12:4590 , piece_id=HP4BRKP7YZJFTSPOUONWDLRIVBZAESSM32YD4X4BA5V4HGU425MA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.067077802 (547h24m45.938160838s) [noise=true , address=45.91.116.5:4297 , piece_id=YO7ASJXSJABEQSQWLVWAID4TLFLS5J5FWCEG5DLFRFTJMH6JIJ4A, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.068479281 (547h24m45.939562316s) [noise=true , address=45.91.116.4:4937 , piece_id=QMRJT4RIWXEJQ4L2YOKUXMP6DI7ICNXNTNOOMA6QJL2SGKDOITWA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.092376034 (547h24m45.963459069s) [noise=true , address=77.90.31.11:4554 , piece_id=RIM3YVQL76KGL74X32UCNKRM7HHFIOUJ2BJZDVAZ6H3V5QKIMKZQ, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.155681188 (547h24m46.026764233s) [noise=true , address=77.90.31.19:4842 , piece_id=JYUC6HPMWUBLFVNQQPM4ZM7J76DHBPXCK25XGSYXDCOOX5OTNMAQ, seg_pos={0 0}, upload=480002]: Done uploading piece. err: <nil> Dec 28 10:53:46.155707277 (547h24m46.026790312s) [seg_pos={0 0}, upload=480002]: Segment reached optimal threshold of 13 pieces. Dec 28 10:53:46.155967233 (547h24m46.027050288s) [noise=true , address=77.90.31.15:4697 , piece_id=TDMVGRIHVTB26UL74JTKP4WS7I6HI3EEWONUAXB7WXZWEPLLDERA, seg_pos={0 0}, upload=480002]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 10:53:46.155976019 (547h24m46.027059064s) [noise=true , address=77.90.31.15:4697 , piece_id=TDMVGRIHVTB26UL74JTKP4WS7I6HI3EEWONUAXB7WXZWEPLLDERA, seg_pos={0 0}, upload=480002]: Long tail cancellation due to optimal threshold Dec 28 10:53:46.156065570 (547h24m46.027148625s) [noise=true , address=45.91.116.14:4190 , piece_id=25FPEMOAC4WWMTDYGB4YKV2BLBBT6VA4BZMM3REGR3C2MOXECBZQ, seg_pos={0 0}, upload=480002]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:54120->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:54120->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54120->45.91.116.14:4190: use of closed network connection Dec 28 10:53:46.156076060 (547h24m46.027159105s) [noise=true , address=45.91.116.14:4190 , piece_id=25FPEMOAC4WWMTDYGB4YKV2BLBBT6VA4BZMM3REGR3C2MOXECBZQ, seg_pos={0 0}, upload=480002]: Long tail cancellation due to optimal threshold Dec 28 10:53:46.156341756 (547h24m46.027424791s) [noise=true , address=45.91.116.12:4228 , piece_id=TUOEAWNBH5HWDA5XOSZHKQL3CQKUE5IOT4VY4AJOQ4T2MQAJHFHQ, seg_pos={0 0}, upload=480002]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 10:53:46.156351585 (547h24m46.02743463s) [noise=true , address=45.91.116.12:4228 , piece_id=TUOEAWNBH5HWDA5XOSZHKQL3CQKUE5IOT4VY4AJOQ4T2MQAJHFHQ, seg_pos={0 0}, upload=480002]: Long tail cancellation due to optimal threshold Dec 28 10:53:46.156379498 (547h24m46.027462533s) [seg_pos={0 0}, upload=480002]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:53:46.156485770 (547h24m46.027568805s) [upload=480002]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:53:46.156492503 (547h24m46.027575548s) [upload=480002]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:53:46.441783576 (547h24m46.312866621s) [upload=480002]: Done uploading segments... Dec 28 10:53:46.441827880 (547h24m46.312910915s) [upload=480002]: Upload finished. err: <nil> Duration: 30.00294255s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..............X..............X.X.........X...XX......XXXX...] duration: 4.867179985s trace: https://jaeger.dev.storj.io/trace/286db894f15472df FAIL (2025-12-28 10:53:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.867204982s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...............X..............X.X.........X...XX......XXXX..] duration: 5.001044835s trace: https://jaeger.dev.storj.io/trace/286db894f15472df FAIL (2025-12-28 10:53:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001070836s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [................X..............X.X.........X...XX......XXXX.] duration: 3.451102685s trace: https://jaeger.dev.storj.io/trace/286db894f15472df FAIL (2025-12-28 10:53:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.451127011s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.........X..XX.....X..X............XX...........XXX.X.X.X...] duration: 6.302704838s trace: https://jaeger.dev.storj.io/trace/435a981a536006d7 FAIL (2025-12-28 10:53:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 10:50:13.399801602 (547h21m13.270884647s) [upload=479956]: Starting upload Dec 28 10:50:13.399865222 (547h21m13.270948267s) [upload=479956]: Done starting upload Dec 28 10:50:13.400196123 (547h21m13.271279168s) [upload=479956]: Uploading segments... Dec 28 10:50:13.400250165 (547h21m13.27133321s) [upload=479956]: Got next segment. Inline: false Dec 28 10:50:13.400293839 (547h21m13.271376884s) [upload=479956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:50:13.400298658 (547h21m13.271381703s) [upload=479956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:50:13.647822776 (547h21m13.518905831s) [seg_pos={0 0}, upload=479956]: Begin upload segment... Dec 28 10:50:13.647887540 (547h21m13.518970585s) [seg_pos={0 0}, upload=479956]: Done begin upload segment. Dec 28 10:50:13.647917727 (547h21m13.519000772s) [noise=true , address=77.90.31.15:4700 , piece_id=6YUNW7FUYB3NS2VESDCIPFDPZ23Q52Q35BPJ72V7HLROCKCTJMJA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.647927716 (547h21m13.519010761s) [noise=true , address=77.90.31.13:4626 , piece_id=QQPFRBUPXGKQQWOOBZKEPVVDJ7BHRHM45FAHKJS6WCMEZIS3KICA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.648010563 (547h21m13.519093608s) [noise=true , address=145.79.227.10:4401 , piece_id=R3JTDKE7JWLRTW2OE44L6KVY4QNRHYKR7Z3FQNBQ63U57RID3MYQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.648705507 (547h21m13.519788562s) [noise=true , address=77.90.31.18:4805 , piece_id=AWQCO23J2Z6B6O4GVEHSIRB6JTQPT6XBTTXRCFGDMUUHRML5ATQQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.649316771 (547h21m13.520399816s) [noise=true , address=45.91.116.14:4192 , piece_id=ZDOLUTM3D6A4DVHCII7ZK64ODY37LQN34TAPBEMUIC62PTKYVXVQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.649917214 (547h21m13.521000259s) [noise=true , address=145.79.227.14:4839 , piece_id=BVKCJ4PVSWAZEFM6KHQARWKV64RVQCCGKTOHBBWUS3C63GA7PZSQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.650609723 (547h21m13.521692768s) [noise=true , address=145.79.227.41:4087 , piece_id=4BJZRQHZKO7ZYRL4OQBQXZ7KEY4BQEZ5OP5ZHZRUZHWZVM3G7ZQQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.651146875 (547h21m13.52222992s) [noise=true , address=45.91.116.5:4297 , piece_id=7DZDGURNWIQYPT6CH65GVXNYF5NTVP2XFJ62JOYCIK54J76AQ3NQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.652788531 (547h21m13.523871576s) [noise=true , address=45.91.116.9:4262 , piece_id=3DXMVN6NF5CIC5WJWPA2J5GKFPHMK4QFYTT4DXYOJESN3IMFIPYA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.653159418 (547h21m13.524242463s) [noise=true , address=145.79.227.11:4731 , piece_id=HSFYDCUWYAOXZFJ5YVQSCDGHVKACHVF6BUI2DOQRMMG6B26SHVOA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.654761608 (547h21m13.525844644s) [noise=true , address=145.79.227.32:4834 , piece_id=N6NPNAFLJ2XXLHOCOBPUUS45KMFEIDCMPEMLLMRHDQDJFCODJM2Q, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.655030860 (547h21m13.526113916s) [noise=true , address=145.79.227.16:4911 , piece_id=67OACSCA7QQZDG6Z77OURXTINFEDUTG4B26QBN2WH3UTI257D6NQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.656387574 (547h21m13.527470609s) [noise=true , address=77.90.31.21:4880 , piece_id=OELXN552ZQRTMGKD4TYD2TYI6YN5DPT7KLRT7ZBTJEK7HKVKBAEA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.656947761 (547h21m13.528030807s) [noise=true , address=145.79.227.23:4547 , piece_id=LIWA5SUEYFPNTFXIQ7KOTDGFHTYS23RH5RQK73LGILNPA6TARESQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.658037116 (547h21m13.529120151s) [noise=true , address=145.79.227.26:4655 , piece_id=KPSLSLXJL7RRVDY2WILJ7M7ZA4CRCBDQWZS5JXPDCWHUFPQHEOPA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.658991172 (547h21m13.530074217s) [noise=true , address=145.79.227.13:4803 , piece_id=OORDYWAVFTKA4T2FEZJQXV5NS46Z4BOOQPFVYG23JAO2FZAXKDWQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:15.281547695 (547h21m15.1526308s) [upload=479956]: Next returned nil segment Dec 28 10:50:15.281567543 (547h21m15.152650608s) [upload=479956]: Waiting for error group managing segments... Dec 28 10:50:15.665057533 (547h21m15.536140568s) [noise=true , address=145.79.227.23:4547 , piece_id=LIWA5SUEYFPNTFXIQ7KOTDGFHTYS23RH5RQK73LGILNPA6TARESQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.665081959 (547h21m15.536165025s) [noise=true , address=145.79.227.41:4087 , piece_id=4BJZRQHZKO7ZYRL4OQBQXZ7KEY4BQEZ5OP5ZHZRUZHWZVM3G7ZQQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.665625995 (547h21m15.53670904s) [noise=true , address=145.79.227.32:4834 , piece_id=N6NPNAFLJ2XXLHOCOBPUUS45KMFEIDCMPEMLLMRHDQDJFCODJM2Q, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.666180471 (547h21m15.537263516s) [noise=true , address=145.79.227.14:4839 , piece_id=BVKCJ4PVSWAZEFM6KHQARWKV64RVQCCGKTOHBBWUS3C63GA7PZSQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.671725168 (547h21m15.542808204s) [noise=true , address=145.79.227.26:4655 , piece_id=KPSLSLXJL7RRVDY2WILJ7M7ZA4CRCBDQWZS5JXPDCWHUFPQHEOPA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.672693372 (547h21m15.543776417s) [noise=true , address=145.79.227.10:4401 , piece_id=R3JTDKE7JWLRTW2OE44L6KVY4QNRHYKR7Z3FQNBQ63U57RID3MYQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.674592569 (547h21m15.545675614s) [noise=true , address=145.79.227.16:4911 , piece_id=67OACSCA7QQZDG6Z77OURXTINFEDUTG4B26QBN2WH3UTI257D6NQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.681223545 (547h21m15.55230658s) [noise=true , address=145.79.227.13:4803 , piece_id=OORDYWAVFTKA4T2FEZJQXV5NS46Z4BOOQPFVYG23JAO2FZAXKDWQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.698829931 (547h21m15.569912976s) [noise=true , address=145.79.227.11:4731 , piece_id=HSFYDCUWYAOXZFJ5YVQSCDGHVKACHVF6BUI2DOQRMMG6B26SHVOA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.899240206 (547h21m15.770323251s) [noise=true , address=45.91.116.14:4192 , piece_id=ZDOLUTM3D6A4DVHCII7ZK64ODY37LQN34TAPBEMUIC62PTKYVXVQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.933158663 (547h21m15.804241718s) [noise=true , address=77.90.31.15:4700 , piece_id=6YUNW7FUYB3NS2VESDCIPFDPZ23Q52Q35BPJ72V7HLROCKCTJMJA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.947939307 (547h21m15.819022352s) [noise=true , address=77.90.31.18:4805 , piece_id=AWQCO23J2Z6B6O4GVEHSIRB6JTQPT6XBTTXRCFGDMUUHRML5ATQQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.980610679 (547h21m15.851693724s) [noise=true , address=45.91.116.5:4297 , piece_id=7DZDGURNWIQYPT6CH65GVXNYF5NTVP2XFJ62JOYCIK54J76AQ3NQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.980636909 (547h21m15.851719954s) [seg_pos={0 0}, upload=479956]: Segment reached optimal threshold of 13 pieces. Dec 28 10:50:15.980837681 (547h21m15.851920726s) [noise=true , address=77.90.31.13:4626 , piece_id=QQPFRBUPXGKQQWOOBZKEPVVDJ7BHRHM45FAHKJS6WCMEZIS3KICA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:60134->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:60134->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60134->77.90.31.13:4626: use of closed network connection Dec 28 10:50:15.980882767 (547h21m15.851965842s) [noise=true , address=77.90.31.13:4626 , piece_id=QQPFRBUPXGKQQWOOBZKEPVVDJ7BHRHM45FAHKJS6WCMEZIS3KICA, seg_pos={0 0}, upload=479956]: Long tail cancellation due to optimal threshold Dec 28 10:50:15.980984791 (547h21m15.852067846s) [noise=true , address=45.91.116.9:4262 , piece_id=3DXMVN6NF5CIC5WJWPA2J5GKFPHMK4QFYTT4DXYOJESN3IMFIPYA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 28 10:50:15.980996052 (547h21m15.852079107s) [noise=true , address=45.91.116.9:4262 , piece_id=3DXMVN6NF5CIC5WJWPA2J5GKFPHMK4QFYTT4DXYOJESN3IMFIPYA, seg_pos={0 0}, upload=479956]: Long tail cancellation due to optimal threshold Dec 28 10:50:15.981217845 (547h21m15.85230089s) [noise=true , address=77.90.31.21:4880 , piece_id=OELXN552ZQRTMGKD4TYD2TYI6YN5DPT7KLRT7ZBTJEK7HKVKBAEA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 28 10:50:15.981224738 (547h21m15.852307773s) [noise=true , address=77.90.31.21:4880 , piece_id=OELXN552ZQRTMGKD4TYD2TYI6YN5DPT7KLRT7ZBTJEK7HKVKBAEA, seg_pos={0 0}, upload=479956]: Long tail cancellation due to optimal threshold Dec 28 10:50:15.981245308 (547h21m15.852328343s) [seg_pos={0 0}, upload=479956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:50:15.981333144 (547h21m15.85241618s) [upload=479956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:50:15.981339677 (547h21m15.852422722s) [upload=479956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:50:16.270627761 (547h21m16.141710796s) [upload=479956]: Done uploading segments... Dec 28 10:50:16.270654691 (547h21m16.141737736s) [upload=479956]: Upload finished. err: <nil> Duration: 6.302741378s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.................X..............X.X.........X...XX......XXXX] duration: 5.001161497s trace: https://jaeger.dev.storj.io/trace/286db894f15472df FAIL (2025-12-28 10:52:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001180874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..........X..XX.....X..X............XX...........XXX.X.X.X..] duration: 5.985895007s trace: https://jaeger.dev.storj.io/trace/435a981a536006d7 FAIL (2025-12-28 10:51:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 10:50:13.399801602 (547h21m13.270884647s) [upload=479956]: Starting upload Dec 28 10:50:13.399865222 (547h21m13.270948267s) [upload=479956]: Done starting upload Dec 28 10:50:13.400196123 (547h21m13.271279168s) [upload=479956]: Uploading segments... Dec 28 10:50:13.400250165 (547h21m13.27133321s) [upload=479956]: Got next segment. Inline: false Dec 28 10:50:13.400293839 (547h21m13.271376884s) [upload=479956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:50:13.400298658 (547h21m13.271381703s) [upload=479956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:50:13.647822776 (547h21m13.518905831s) [seg_pos={0 0}, upload=479956]: Begin upload segment... Dec 28 10:50:13.647887540 (547h21m13.518970585s) [seg_pos={0 0}, upload=479956]: Done begin upload segment. Dec 28 10:50:13.647917727 (547h21m13.519000772s) [noise=true , address=77.90.31.15:4700 , piece_id=6YUNW7FUYB3NS2VESDCIPFDPZ23Q52Q35BPJ72V7HLROCKCTJMJA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.647927716 (547h21m13.519010761s) [noise=true , address=77.90.31.13:4626 , piece_id=QQPFRBUPXGKQQWOOBZKEPVVDJ7BHRHM45FAHKJS6WCMEZIS3KICA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.648010563 (547h21m13.519093608s) [noise=true , address=145.79.227.10:4401 , piece_id=R3JTDKE7JWLRTW2OE44L6KVY4QNRHYKR7Z3FQNBQ63U57RID3MYQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.648705507 (547h21m13.519788562s) [noise=true , address=77.90.31.18:4805 , piece_id=AWQCO23J2Z6B6O4GVEHSIRB6JTQPT6XBTTXRCFGDMUUHRML5ATQQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.649316771 (547h21m13.520399816s) [noise=true , address=45.91.116.14:4192 , piece_id=ZDOLUTM3D6A4DVHCII7ZK64ODY37LQN34TAPBEMUIC62PTKYVXVQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.649917214 (547h21m13.521000259s) [noise=true , address=145.79.227.14:4839 , piece_id=BVKCJ4PVSWAZEFM6KHQARWKV64RVQCCGKTOHBBWUS3C63GA7PZSQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.650609723 (547h21m13.521692768s) [noise=true , address=145.79.227.41:4087 , piece_id=4BJZRQHZKO7ZYRL4OQBQXZ7KEY4BQEZ5OP5ZHZRUZHWZVM3G7ZQQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.651146875 (547h21m13.52222992s) [noise=true , address=45.91.116.5:4297 , piece_id=7DZDGURNWIQYPT6CH65GVXNYF5NTVP2XFJ62JOYCIK54J76AQ3NQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.652788531 (547h21m13.523871576s) [noise=true , address=45.91.116.9:4262 , piece_id=3DXMVN6NF5CIC5WJWPA2J5GKFPHMK4QFYTT4DXYOJESN3IMFIPYA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.653159418 (547h21m13.524242463s) [noise=true , address=145.79.227.11:4731 , piece_id=HSFYDCUWYAOXZFJ5YVQSCDGHVKACHVF6BUI2DOQRMMG6B26SHVOA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.654761608 (547h21m13.525844644s) [noise=true , address=145.79.227.32:4834 , piece_id=N6NPNAFLJ2XXLHOCOBPUUS45KMFEIDCMPEMLLMRHDQDJFCODJM2Q, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.655030860 (547h21m13.526113916s) [noise=true , address=145.79.227.16:4911 , piece_id=67OACSCA7QQZDG6Z77OURXTINFEDUTG4B26QBN2WH3UTI257D6NQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.656387574 (547h21m13.527470609s) [noise=true , address=77.90.31.21:4880 , piece_id=OELXN552ZQRTMGKD4TYD2TYI6YN5DPT7KLRT7ZBTJEK7HKVKBAEA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.656947761 (547h21m13.528030807s) [noise=true , address=145.79.227.23:4547 , piece_id=LIWA5SUEYFPNTFXIQ7KOTDGFHTYS23RH5RQK73LGILNPA6TARESQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.658037116 (547h21m13.529120151s) [noise=true , address=145.79.227.26:4655 , piece_id=KPSLSLXJL7RRVDY2WILJ7M7ZA4CRCBDQWZS5JXPDCWHUFPQHEOPA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.658991172 (547h21m13.530074217s) [noise=true , address=145.79.227.13:4803 , piece_id=OORDYWAVFTKA4T2FEZJQXV5NS46Z4BOOQPFVYG23JAO2FZAXKDWQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:15.281547695 (547h21m15.1526308s) [upload=479956]: Next returned nil segment Dec 28 10:50:15.281567543 (547h21m15.152650608s) [upload=479956]: Waiting for error group managing segments... Dec 28 10:50:15.665057533 (547h21m15.536140568s) [noise=true , address=145.79.227.23:4547 , piece_id=LIWA5SUEYFPNTFXIQ7KOTDGFHTYS23RH5RQK73LGILNPA6TARESQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.665081959 (547h21m15.536165025s) [noise=true , address=145.79.227.41:4087 , piece_id=4BJZRQHZKO7ZYRL4OQBQXZ7KEY4BQEZ5OP5ZHZRUZHWZVM3G7ZQQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.665625995 (547h21m15.53670904s) [noise=true , address=145.79.227.32:4834 , piece_id=N6NPNAFLJ2XXLHOCOBPUUS45KMFEIDCMPEMLLMRHDQDJFCODJM2Q, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.666180471 (547h21m15.537263516s) [noise=true , address=145.79.227.14:4839 , piece_id=BVKCJ4PVSWAZEFM6KHQARWKV64RVQCCGKTOHBBWUS3C63GA7PZSQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.671725168 (547h21m15.542808204s) [noise=true , address=145.79.227.26:4655 , piece_id=KPSLSLXJL7RRVDY2WILJ7M7ZA4CRCBDQWZS5JXPDCWHUFPQHEOPA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.672693372 (547h21m15.543776417s) [noise=true , address=145.79.227.10:4401 , piece_id=R3JTDKE7JWLRTW2OE44L6KVY4QNRHYKR7Z3FQNBQ63U57RID3MYQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.674592569 (547h21m15.545675614s) [noise=true , address=145.79.227.16:4911 , piece_id=67OACSCA7QQZDG6Z77OURXTINFEDUTG4B26QBN2WH3UTI257D6NQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.681223545 (547h21m15.55230658s) [noise=true , address=145.79.227.13:4803 , piece_id=OORDYWAVFTKA4T2FEZJQXV5NS46Z4BOOQPFVYG23JAO2FZAXKDWQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.698829931 (547h21m15.569912976s) [noise=true , address=145.79.227.11:4731 , piece_id=HSFYDCUWYAOXZFJ5YVQSCDGHVKACHVF6BUI2DOQRMMG6B26SHVOA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.899240206 (547h21m15.770323251s) [noise=true , address=45.91.116.14:4192 , piece_id=ZDOLUTM3D6A4DVHCII7ZK64ODY37LQN34TAPBEMUIC62PTKYVXVQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.933158663 (547h21m15.804241718s) [noise=true , address=77.90.31.15:4700 , piece_id=6YUNW7FUYB3NS2VESDCIPFDPZ23Q52Q35BPJ72V7HLROCKCTJMJA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.947939307 (547h21m15.819022352s) [noise=true , address=77.90.31.18:4805 , piece_id=AWQCO23J2Z6B6O4GVEHSIRB6JTQPT6XBTTXRCFGDMUUHRML5ATQQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.980610679 (547h21m15.851693724s) [noise=true , address=45.91.116.5:4297 , piece_id=7DZDGURNWIQYPT6CH65GVXNYF5NTVP2XFJ62JOYCIK54J76AQ3NQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.980636909 (547h21m15.851719954s) [seg_pos={0 0}, upload=479956]: Segment reached optimal threshold of 13 pieces. Dec 28 10:50:15.980837681 (547h21m15.851920726s) [noise=true , address=77.90.31.13:4626 , piece_id=QQPFRBUPXGKQQWOOBZKEPVVDJ7BHRHM45FAHKJS6WCMEZIS3KICA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:60134->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:60134->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60134->77.90.31.13:4626: use of closed network connection Dec 28 10:50:15.980882767 (547h21m15.851965842s) [noise=true , address=77.90.31.13:4626 , piece_id=QQPFRBUPXGKQQWOOBZKEPVVDJ7BHRHM45FAHKJS6WCMEZIS3KICA, seg_pos={0 0}, upload=479956]: Long tail cancellation due to optimal threshold Dec 28 10:50:15.980984791 (547h21m15.852067846s) [noise=true , address=45.91.116.9:4262 , piece_id=3DXMVN6NF5CIC5WJWPA2J5GKFPHMK4QFYTT4DXYOJESN3IMFIPYA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 28 10:50:15.980996052 (547h21m15.852079107s) [noise=true , address=45.91.116.9:4262 , piece_id=3DXMVN6NF5CIC5WJWPA2J5GKFPHMK4QFYTT4DXYOJESN3IMFIPYA, seg_pos={0 0}, upload=479956]: Long tail cancellation due to optimal threshold Dec 28 10:50:15.981217845 (547h21m15.85230089s) [noise=true , address=77.90.31.21:4880 , piece_id=OELXN552ZQRTMGKD4TYD2TYI6YN5DPT7KLRT7ZBTJEK7HKVKBAEA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 28 10:50:15.981224738 (547h21m15.852307773s) [noise=true , address=77.90.31.21:4880 , piece_id=OELXN552ZQRTMGKD4TYD2TYI6YN5DPT7KLRT7ZBTJEK7HKVKBAEA, seg_pos={0 0}, upload=479956]: Long tail cancellation due to optimal threshold Dec 28 10:50:15.981245308 (547h21m15.852328343s) [seg_pos={0 0}, upload=479956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:50:15.981333144 (547h21m15.85241618s) [upload=479956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:50:15.981339677 (547h21m15.852422722s) [upload=479956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:50:16.270627761 (547h21m16.141710796s) [upload=479956]: Done uploading segments... Dec 28 10:50:16.270654691 (547h21m16.141737736s) [upload=479956]: Upload finished. err: <nil> Duration: 5.985924854s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..................X..............X.X.........X...XX......XXX] duration: 5.000795153s trace: https://jaeger.dev.storj.io/trace/56884947554ab075 FAIL (2025-12-28 10:51:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000821192s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...........X..XX.....X..X............XX...........XXX.X.X.X.] duration: 6.383691951s trace: https://jaeger.dev.storj.io/trace/435a981a536006d7 FAIL (2025-12-28 10:51:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 10:50:13.399801602 (547h21m13.270884647s) [upload=479956]: Starting upload Dec 28 10:50:13.399865222 (547h21m13.270948267s) [upload=479956]: Done starting upload Dec 28 10:50:13.400196123 (547h21m13.271279168s) [upload=479956]: Uploading segments... Dec 28 10:50:13.400250165 (547h21m13.27133321s) [upload=479956]: Got next segment. Inline: false Dec 28 10:50:13.400293839 (547h21m13.271376884s) [upload=479956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:50:13.400298658 (547h21m13.271381703s) [upload=479956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:50:13.647822776 (547h21m13.518905831s) [seg_pos={0 0}, upload=479956]: Begin upload segment... Dec 28 10:50:13.647887540 (547h21m13.518970585s) [seg_pos={0 0}, upload=479956]: Done begin upload segment. Dec 28 10:50:13.647917727 (547h21m13.519000772s) [noise=true , address=77.90.31.15:4700 , piece_id=6YUNW7FUYB3NS2VESDCIPFDPZ23Q52Q35BPJ72V7HLROCKCTJMJA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.647927716 (547h21m13.519010761s) [noise=true , address=77.90.31.13:4626 , piece_id=QQPFRBUPXGKQQWOOBZKEPVVDJ7BHRHM45FAHKJS6WCMEZIS3KICA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.648010563 (547h21m13.519093608s) [noise=true , address=145.79.227.10:4401 , piece_id=R3JTDKE7JWLRTW2OE44L6KVY4QNRHYKR7Z3FQNBQ63U57RID3MYQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.648705507 (547h21m13.519788562s) [noise=true , address=77.90.31.18:4805 , piece_id=AWQCO23J2Z6B6O4GVEHSIRB6JTQPT6XBTTXRCFGDMUUHRML5ATQQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.649316771 (547h21m13.520399816s) [noise=true , address=45.91.116.14:4192 , piece_id=ZDOLUTM3D6A4DVHCII7ZK64ODY37LQN34TAPBEMUIC62PTKYVXVQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.649917214 (547h21m13.521000259s) [noise=true , address=145.79.227.14:4839 , piece_id=BVKCJ4PVSWAZEFM6KHQARWKV64RVQCCGKTOHBBWUS3C63GA7PZSQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.650609723 (547h21m13.521692768s) [noise=true , address=145.79.227.41:4087 , piece_id=4BJZRQHZKO7ZYRL4OQBQXZ7KEY4BQEZ5OP5ZHZRUZHWZVM3G7ZQQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.651146875 (547h21m13.52222992s) [noise=true , address=45.91.116.5:4297 , piece_id=7DZDGURNWIQYPT6CH65GVXNYF5NTVP2XFJ62JOYCIK54J76AQ3NQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.652788531 (547h21m13.523871576s) [noise=true , address=45.91.116.9:4262 , piece_id=3DXMVN6NF5CIC5WJWPA2J5GKFPHMK4QFYTT4DXYOJESN3IMFIPYA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.653159418 (547h21m13.524242463s) [noise=true , address=145.79.227.11:4731 , piece_id=HSFYDCUWYAOXZFJ5YVQSCDGHVKACHVF6BUI2DOQRMMG6B26SHVOA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.654761608 (547h21m13.525844644s) [noise=true , address=145.79.227.32:4834 , piece_id=N6NPNAFLJ2XXLHOCOBPUUS45KMFEIDCMPEMLLMRHDQDJFCODJM2Q, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.655030860 (547h21m13.526113916s) [noise=true , address=145.79.227.16:4911 , piece_id=67OACSCA7QQZDG6Z77OURXTINFEDUTG4B26QBN2WH3UTI257D6NQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.656387574 (547h21m13.527470609s) [noise=true , address=77.90.31.21:4880 , piece_id=OELXN552ZQRTMGKD4TYD2TYI6YN5DPT7KLRT7ZBTJEK7HKVKBAEA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.656947761 (547h21m13.528030807s) [noise=true , address=145.79.227.23:4547 , piece_id=LIWA5SUEYFPNTFXIQ7KOTDGFHTYS23RH5RQK73LGILNPA6TARESQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.658037116 (547h21m13.529120151s) [noise=true , address=145.79.227.26:4655 , piece_id=KPSLSLXJL7RRVDY2WILJ7M7ZA4CRCBDQWZS5JXPDCWHUFPQHEOPA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.658991172 (547h21m13.530074217s) [noise=true , address=145.79.227.13:4803 , piece_id=OORDYWAVFTKA4T2FEZJQXV5NS46Z4BOOQPFVYG23JAO2FZAXKDWQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:15.281547695 (547h21m15.1526308s) [upload=479956]: Next returned nil segment Dec 28 10:50:15.281567543 (547h21m15.152650608s) [upload=479956]: Waiting for error group managing segments... Dec 28 10:50:15.665057533 (547h21m15.536140568s) [noise=true , address=145.79.227.23:4547 , piece_id=LIWA5SUEYFPNTFXIQ7KOTDGFHTYS23RH5RQK73LGILNPA6TARESQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.665081959 (547h21m15.536165025s) [noise=true , address=145.79.227.41:4087 , piece_id=4BJZRQHZKO7ZYRL4OQBQXZ7KEY4BQEZ5OP5ZHZRUZHWZVM3G7ZQQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.665625995 (547h21m15.53670904s) [noise=true , address=145.79.227.32:4834 , piece_id=N6NPNAFLJ2XXLHOCOBPUUS45KMFEIDCMPEMLLMRHDQDJFCODJM2Q, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.666180471 (547h21m15.537263516s) [noise=true , address=145.79.227.14:4839 , piece_id=BVKCJ4PVSWAZEFM6KHQARWKV64RVQCCGKTOHBBWUS3C63GA7PZSQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.671725168 (547h21m15.542808204s) [noise=true , address=145.79.227.26:4655 , piece_id=KPSLSLXJL7RRVDY2WILJ7M7ZA4CRCBDQWZS5JXPDCWHUFPQHEOPA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.672693372 (547h21m15.543776417s) [noise=true , address=145.79.227.10:4401 , piece_id=R3JTDKE7JWLRTW2OE44L6KVY4QNRHYKR7Z3FQNBQ63U57RID3MYQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.674592569 (547h21m15.545675614s) [noise=true , address=145.79.227.16:4911 , piece_id=67OACSCA7QQZDG6Z77OURXTINFEDUTG4B26QBN2WH3UTI257D6NQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.681223545 (547h21m15.55230658s) [noise=true , address=145.79.227.13:4803 , piece_id=OORDYWAVFTKA4T2FEZJQXV5NS46Z4BOOQPFVYG23JAO2FZAXKDWQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.698829931 (547h21m15.569912976s) [noise=true , address=145.79.227.11:4731 , piece_id=HSFYDCUWYAOXZFJ5YVQSCDGHVKACHVF6BUI2DOQRMMG6B26SHVOA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.899240206 (547h21m15.770323251s) [noise=true , address=45.91.116.14:4192 , piece_id=ZDOLUTM3D6A4DVHCII7ZK64ODY37LQN34TAPBEMUIC62PTKYVXVQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.933158663 (547h21m15.804241718s) [noise=true , address=77.90.31.15:4700 , piece_id=6YUNW7FUYB3NS2VESDCIPFDPZ23Q52Q35BPJ72V7HLROCKCTJMJA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.947939307 (547h21m15.819022352s) [noise=true , address=77.90.31.18:4805 , piece_id=AWQCO23J2Z6B6O4GVEHSIRB6JTQPT6XBTTXRCFGDMUUHRML5ATQQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.980610679 (547h21m15.851693724s) [noise=true , address=45.91.116.5:4297 , piece_id=7DZDGURNWIQYPT6CH65GVXNYF5NTVP2XFJ62JOYCIK54J76AQ3NQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.980636909 (547h21m15.851719954s) [seg_pos={0 0}, upload=479956]: Segment reached optimal threshold of 13 pieces. Dec 28 10:50:15.980837681 (547h21m15.851920726s) [noise=true , address=77.90.31.13:4626 , piece_id=QQPFRBUPXGKQQWOOBZKEPVVDJ7BHRHM45FAHKJS6WCMEZIS3KICA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:60134->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:60134->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60134->77.90.31.13:4626: use of closed network connection Dec 28 10:50:15.980882767 (547h21m15.851965842s) [noise=true , address=77.90.31.13:4626 , piece_id=QQPFRBUPXGKQQWOOBZKEPVVDJ7BHRHM45FAHKJS6WCMEZIS3KICA, seg_pos={0 0}, upload=479956]: Long tail cancellation due to optimal threshold Dec 28 10:50:15.980984791 (547h21m15.852067846s) [noise=true , address=45.91.116.9:4262 , piece_id=3DXMVN6NF5CIC5WJWPA2J5GKFPHMK4QFYTT4DXYOJESN3IMFIPYA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 28 10:50:15.980996052 (547h21m15.852079107s) [noise=true , address=45.91.116.9:4262 , piece_id=3DXMVN6NF5CIC5WJWPA2J5GKFPHMK4QFYTT4DXYOJESN3IMFIPYA, seg_pos={0 0}, upload=479956]: Long tail cancellation due to optimal threshold Dec 28 10:50:15.981217845 (547h21m15.85230089s) [noise=true , address=77.90.31.21:4880 , piece_id=OELXN552ZQRTMGKD4TYD2TYI6YN5DPT7KLRT7ZBTJEK7HKVKBAEA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 28 10:50:15.981224738 (547h21m15.852307773s) [noise=true , address=77.90.31.21:4880 , piece_id=OELXN552ZQRTMGKD4TYD2TYI6YN5DPT7KLRT7ZBTJEK7HKVKBAEA, seg_pos={0 0}, upload=479956]: Long tail cancellation due to optimal threshold Dec 28 10:50:15.981245308 (547h21m15.852328343s) [seg_pos={0 0}, upload=479956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:50:15.981333144 (547h21m15.85241618s) [upload=479956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:50:15.981339677 (547h21m15.852422722s) [upload=479956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:50:16.270627761 (547h21m16.141710796s) [upload=479956]: Done uploading segments... Dec 28 10:50:16.270654691 (547h21m16.141737736s) [upload=479956]: Upload finished. err: <nil> Duration: 6.38372186s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [X..................X..............X.X.........X...XX......XX] duration: 5.0017669s trace: https://jaeger.dev.storj.io/trace/5a2bae3f6caf8cfd FAIL (2025-12-28 10:50:46 UTC) orbiter warning: context deadline exceeded Duration: 5.00178785s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [............X..XX.....X..X............XX...........XXX.X.X.X] duration: 30.002879662s trace: https://jaeger.dev.storj.io/trace/435a981a536006d7 FAIL (2025-12-28 10:50:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 10:50:13.399801602 (547h21m13.270884647s) [upload=479956]: Starting upload Dec 28 10:50:13.399865222 (547h21m13.270948267s) [upload=479956]: Done starting upload Dec 28 10:50:13.400196123 (547h21m13.271279168s) [upload=479956]: Uploading segments... Dec 28 10:50:13.400250165 (547h21m13.27133321s) [upload=479956]: Got next segment. Inline: false Dec 28 10:50:13.400293839 (547h21m13.271376884s) [upload=479956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:50:13.400298658 (547h21m13.271381703s) [upload=479956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:50:13.647822776 (547h21m13.518905831s) [seg_pos={0 0}, upload=479956]: Begin upload segment... Dec 28 10:50:13.647887540 (547h21m13.518970585s) [seg_pos={0 0}, upload=479956]: Done begin upload segment. Dec 28 10:50:13.647917727 (547h21m13.519000772s) [noise=true , address=77.90.31.15:4700 , piece_id=6YUNW7FUYB3NS2VESDCIPFDPZ23Q52Q35BPJ72V7HLROCKCTJMJA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.647927716 (547h21m13.519010761s) [noise=true , address=77.90.31.13:4626 , piece_id=QQPFRBUPXGKQQWOOBZKEPVVDJ7BHRHM45FAHKJS6WCMEZIS3KICA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.648010563 (547h21m13.519093608s) [noise=true , address=145.79.227.10:4401 , piece_id=R3JTDKE7JWLRTW2OE44L6KVY4QNRHYKR7Z3FQNBQ63U57RID3MYQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.648705507 (547h21m13.519788562s) [noise=true , address=77.90.31.18:4805 , piece_id=AWQCO23J2Z6B6O4GVEHSIRB6JTQPT6XBTTXRCFGDMUUHRML5ATQQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.649316771 (547h21m13.520399816s) [noise=true , address=45.91.116.14:4192 , piece_id=ZDOLUTM3D6A4DVHCII7ZK64ODY37LQN34TAPBEMUIC62PTKYVXVQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.649917214 (547h21m13.521000259s) [noise=true , address=145.79.227.14:4839 , piece_id=BVKCJ4PVSWAZEFM6KHQARWKV64RVQCCGKTOHBBWUS3C63GA7PZSQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.650609723 (547h21m13.521692768s) [noise=true , address=145.79.227.41:4087 , piece_id=4BJZRQHZKO7ZYRL4OQBQXZ7KEY4BQEZ5OP5ZHZRUZHWZVM3G7ZQQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.651146875 (547h21m13.52222992s) [noise=true , address=45.91.116.5:4297 , piece_id=7DZDGURNWIQYPT6CH65GVXNYF5NTVP2XFJ62JOYCIK54J76AQ3NQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.652788531 (547h21m13.523871576s) [noise=true , address=45.91.116.9:4262 , piece_id=3DXMVN6NF5CIC5WJWPA2J5GKFPHMK4QFYTT4DXYOJESN3IMFIPYA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.653159418 (547h21m13.524242463s) [noise=true , address=145.79.227.11:4731 , piece_id=HSFYDCUWYAOXZFJ5YVQSCDGHVKACHVF6BUI2DOQRMMG6B26SHVOA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.654761608 (547h21m13.525844644s) [noise=true , address=145.79.227.32:4834 , piece_id=N6NPNAFLJ2XXLHOCOBPUUS45KMFEIDCMPEMLLMRHDQDJFCODJM2Q, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.655030860 (547h21m13.526113916s) [noise=true , address=145.79.227.16:4911 , piece_id=67OACSCA7QQZDG6Z77OURXTINFEDUTG4B26QBN2WH3UTI257D6NQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.656387574 (547h21m13.527470609s) [noise=true , address=77.90.31.21:4880 , piece_id=OELXN552ZQRTMGKD4TYD2TYI6YN5DPT7KLRT7ZBTJEK7HKVKBAEA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.656947761 (547h21m13.528030807s) [noise=true , address=145.79.227.23:4547 , piece_id=LIWA5SUEYFPNTFXIQ7KOTDGFHTYS23RH5RQK73LGILNPA6TARESQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.658037116 (547h21m13.529120151s) [noise=true , address=145.79.227.26:4655 , piece_id=KPSLSLXJL7RRVDY2WILJ7M7ZA4CRCBDQWZS5JXPDCWHUFPQHEOPA, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:13.658991172 (547h21m13.530074217s) [noise=true , address=145.79.227.13:4803 , piece_id=OORDYWAVFTKA4T2FEZJQXV5NS46Z4BOOQPFVYG23JAO2FZAXKDWQ, seg_pos={0 0}, upload=479956]: Uploading piece... Dec 28 10:50:15.281547695 (547h21m15.1526308s) [upload=479956]: Next returned nil segment Dec 28 10:50:15.281567543 (547h21m15.152650608s) [upload=479956]: Waiting for error group managing segments... Dec 28 10:50:15.665057533 (547h21m15.536140568s) [noise=true , address=145.79.227.23:4547 , piece_id=LIWA5SUEYFPNTFXIQ7KOTDGFHTYS23RH5RQK73LGILNPA6TARESQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.665081959 (547h21m15.536165025s) [noise=true , address=145.79.227.41:4087 , piece_id=4BJZRQHZKO7ZYRL4OQBQXZ7KEY4BQEZ5OP5ZHZRUZHWZVM3G7ZQQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.665625995 (547h21m15.53670904s) [noise=true , address=145.79.227.32:4834 , piece_id=N6NPNAFLJ2XXLHOCOBPUUS45KMFEIDCMPEMLLMRHDQDJFCODJM2Q, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.666180471 (547h21m15.537263516s) [noise=true , address=145.79.227.14:4839 , piece_id=BVKCJ4PVSWAZEFM6KHQARWKV64RVQCCGKTOHBBWUS3C63GA7PZSQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.671725168 (547h21m15.542808204s) [noise=true , address=145.79.227.26:4655 , piece_id=KPSLSLXJL7RRVDY2WILJ7M7ZA4CRCBDQWZS5JXPDCWHUFPQHEOPA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.672693372 (547h21m15.543776417s) [noise=true , address=145.79.227.10:4401 , piece_id=R3JTDKE7JWLRTW2OE44L6KVY4QNRHYKR7Z3FQNBQ63U57RID3MYQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.674592569 (547h21m15.545675614s) [noise=true , address=145.79.227.16:4911 , piece_id=67OACSCA7QQZDG6Z77OURXTINFEDUTG4B26QBN2WH3UTI257D6NQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.681223545 (547h21m15.55230658s) [noise=true , address=145.79.227.13:4803 , piece_id=OORDYWAVFTKA4T2FEZJQXV5NS46Z4BOOQPFVYG23JAO2FZAXKDWQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.698829931 (547h21m15.569912976s) [noise=true , address=145.79.227.11:4731 , piece_id=HSFYDCUWYAOXZFJ5YVQSCDGHVKACHVF6BUI2DOQRMMG6B26SHVOA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.899240206 (547h21m15.770323251s) [noise=true , address=45.91.116.14:4192 , piece_id=ZDOLUTM3D6A4DVHCII7ZK64ODY37LQN34TAPBEMUIC62PTKYVXVQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.933158663 (547h21m15.804241718s) [noise=true , address=77.90.31.15:4700 , piece_id=6YUNW7FUYB3NS2VESDCIPFDPZ23Q52Q35BPJ72V7HLROCKCTJMJA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.947939307 (547h21m15.819022352s) [noise=true , address=77.90.31.18:4805 , piece_id=AWQCO23J2Z6B6O4GVEHSIRB6JTQPT6XBTTXRCFGDMUUHRML5ATQQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.980610679 (547h21m15.851693724s) [noise=true , address=45.91.116.5:4297 , piece_id=7DZDGURNWIQYPT6CH65GVXNYF5NTVP2XFJ62JOYCIK54J76AQ3NQ, seg_pos={0 0}, upload=479956]: Done uploading piece. err: <nil> Dec 28 10:50:15.980636909 (547h21m15.851719954s) [seg_pos={0 0}, upload=479956]: Segment reached optimal threshold of 13 pieces. Dec 28 10:50:15.980837681 (547h21m15.851920726s) [noise=true , address=77.90.31.13:4626 , piece_id=QQPFRBUPXGKQQWOOBZKEPVVDJ7BHRHM45FAHKJS6WCMEZIS3KICA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:60134->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:60134->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60134->77.90.31.13:4626: use of closed network connection Dec 28 10:50:15.980882767 (547h21m15.851965842s) [noise=true , address=77.90.31.13:4626 , piece_id=QQPFRBUPXGKQQWOOBZKEPVVDJ7BHRHM45FAHKJS6WCMEZIS3KICA, seg_pos={0 0}, upload=479956]: Long tail cancellation due to optimal threshold Dec 28 10:50:15.980984791 (547h21m15.852067846s) [noise=true , address=45.91.116.9:4262 , piece_id=3DXMVN6NF5CIC5WJWPA2J5GKFPHMK4QFYTT4DXYOJESN3IMFIPYA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 28 10:50:15.980996052 (547h21m15.852079107s) [noise=true , address=45.91.116.9:4262 , piece_id=3DXMVN6NF5CIC5WJWPA2J5GKFPHMK4QFYTT4DXYOJESN3IMFIPYA, seg_pos={0 0}, upload=479956]: Long tail cancellation due to optimal threshold Dec 28 10:50:15.981217845 (547h21m15.85230089s) [noise=true , address=77.90.31.21:4880 , piece_id=OELXN552ZQRTMGKD4TYD2TYI6YN5DPT7KLRT7ZBTJEK7HKVKBAEA, seg_pos={0 0}, upload=479956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 28 10:50:15.981224738 (547h21m15.852307773s) [noise=true , address=77.90.31.21:4880 , piece_id=OELXN552ZQRTMGKD4TYD2TYI6YN5DPT7KLRT7ZBTJEK7HKVKBAEA, seg_pos={0 0}, upload=479956]: Long tail cancellation due to optimal threshold Dec 28 10:50:15.981245308 (547h21m15.852328343s) [seg_pos={0 0}, upload=479956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:50:15.981333144 (547h21m15.85241618s) [upload=479956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:50:15.981339677 (547h21m15.852422722s) [upload=479956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:50:16.270627761 (547h21m16.141710796s) [upload=479956]: Done uploading segments... Dec 28 10:50:16.270654691 (547h21m16.141737736s) [upload=479956]: Upload finished. err: <nil> Duration: 30.002902446s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.X..................X..............X.X.........X...XX......X] duration: 5.000940026s trace: https://jaeger.dev.storj.io/trace/738ec5eb1ad5652a FAIL (2025-12-28 10:50:01 UTC) orbiter warning: context deadline exceeded Duration: 5.000961778s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..X..................X..............X.X.........X...XX......] duration: 5.000694589s trace: https://jaeger.dev.storj.io/trace/65dc07c66ac2a58e FAIL (2025-12-28 10:49:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000714927s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.............X..XX.....X..X............XX...........XXX.X.X.] duration: 5.987376727s trace: https://jaeger.dev.storj.io/trace/5451b11343dd2f42 FAIL (2025-12-28 10:49:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3CU6BAIWEZA5UGH73H6ZS3MCM7P7RLDMLBCA7ZUW557ZXZ7AS5VQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: GQ2WUQWMZB45QGWF2MIKRQSGRKJBWZEABT3Q6GO457ZWL5XHCTGQ) context deadline exceeded ======LOGS====== Dec 28 10:48:06.310148263 (547h19m6.181231298s) [upload=479924]: Starting upload Dec 28 10:48:06.310369455 (547h19m6.18145249s) [upload=479924]: Done starting upload Dec 28 10:48:06.310420581 (547h19m6.181503617s) [upload=479924]: Uploading segments... Dec 28 10:48:06.310547103 (547h19m6.181630138s) [upload=479924]: Got next segment. Inline: false Dec 28 10:48:06.310576488 (547h19m6.181659533s) [upload=479924]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:48:06.310582630 (547h19m6.181665665s) [upload=479924]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:48:06.553163324 (547h19m6.424246359s) [seg_pos={0 0}, upload=479924]: Begin upload segment... Dec 28 10:48:06.553214903 (547h19m6.424297948s) [seg_pos={0 0}, upload=479924]: Done begin upload segment. Dec 28 10:48:06.553245691 (547h19m6.424328726s) [noise=true , address=145.79.227.7:4295 , piece_id=JVCXM5X3G75GGCJVUIAJZU5KRAP4N6UDHX5YWWFKLQ2COIQKAIGQ, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.553391970 (547h19m6.424475005s) [noise=true , address=77.90.31.11:4554 , piece_id=F3MQ42IURAHOGVIBXHSEMJBFZYRCUJ7PGEMRO6KFBUZKDHM33FXQ, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.553921238 (547h19m6.425004273s) [noise=true , address=145.79.227.36:4517 , piece_id=PKXX6X2Z3R6QPS2O4V472KLCNNL3H7BFYZFFCPC6CQIQXIZUHYRQ, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.554222943 (547h19m6.425305988s) [noise=true , address=45.91.116.8:4048 , piece_id=3CU6BAIWEZA5UGH73H6ZS3MCM7P7RLDMLBCA7ZUW557ZXZ7AS5VQ, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.554625168 (547h19m6.425708213s) [noise=true , address=145.79.227.46:4603 , piece_id=57GZZIEBM2NTADYLVKBICNSJZSHSUPZYHWOSO5A4NXSTW6XGZYIA, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.554914278 (547h19m6.425997313s) [noise=true , address=145.79.227.48:4301 , piece_id=ZE33WA3F3VYZR6DATAHUMDX7ZBUNTJEH6AWPGDUB7CAQOTMIPS7A, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.555198861 (547h19m6.426281926s) [noise=true , address=45.91.116.5:4297 , piece_id=GQ2WUQWMZB45QGWF2MIKRQSGRKJBWZEABT3Q6GO457ZWL5XHCTGQ, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.555539799 (547h19m6.426622834s) [noise=true , address=145.79.227.51:4410 , piece_id=OZCNZBFE4523TF3LB2B65PQZAWO4NQCJE7H3WWIO2OQS2MSYZFQA, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.555907600 (547h19m6.426990635s) [noise=true , address=145.79.227.45:4230 , piece_id=Z2VO4GCNORLXRSYL3BHUEJEYPCRGAPWOJ4XLIUJ4YCVDSYWU4ACQ, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.556957269 (547h19m6.428040314s) [noise=true , address=145.79.227.42:4123 , piece_id=DAR6KC6JDNLEZWPCZ6UAOZI3THX7BPLS6LFWWOCPB73DWYHYT4TA, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.558083172 (547h19m6.429166217s) [noise=true , address=45.91.116.11:4119 , piece_id=S3ZWJI47R6LRWHRJY2FVQHBNKHHQNPK62KZG3ZXLV45DBRMKLLVA, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.559616001 (547h19m6.430699046s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOHBYMFZQW656MI2HSQMHV5TCFXJ7FJXVNANAKJUBEO6YUD5GPZQ, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.561153680 (547h19m6.432236725s) [noise=true , address=145.79.227.16:4910 , piece_id=K3ZYPYEWS5JMATWKIU3YPJA2M3MOTQDB3FQGTFP3SKQPR2CQE4OQ, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.562976310 (547h19m6.434059345s) [noise=true , address=145.79.227.11:4732 , piece_id=25N43O4KVYRUB2H5TQE2KODL2RJXJLW7RJCTRBCMEWWTQK35I73Q, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.564075544 (547h19m6.435158579s) [noise=true , address=145.79.227.17:4971 , piece_id=JNM5P4N6TXORYJOKABNTFYNN5LDHDHCCRXQNCU57DQC3JGOSM4RQ, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.565284405 (547h19m6.43636744s) [noise=true , address=45.91.116.14:4190 , piece_id=N6QHIKT4FIVGCS2RIHYSMVYXXZ4JNAW5NZ2WIBZKA7KSGPATJDKA, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:08.263029901 (547h19m8.134112936s) [upload=479924]: Next returned nil segment Dec 28 10:48:08.263056782 (547h19m8.134139817s) [upload=479924]: Waiting for error group managing segments... Dec 28 10:48:08.546196217 (547h19m8.417279252s) [noise=true , address=145.79.227.46:4603 , piece_id=57GZZIEBM2NTADYLVKBICNSJZSHSUPZYHWOSO5A4NXSTW6XGZYIA, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.560499743 (547h19m8.431582778s) [noise=true , address=145.79.227.45:4230 , piece_id=Z2VO4GCNORLXRSYL3BHUEJEYPCRGAPWOJ4XLIUJ4YCVDSYWU4ACQ, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.563591801 (547h19m8.434674846s) [noise=true , address=145.79.227.51:4410 , piece_id=OZCNZBFE4523TF3LB2B65PQZAWO4NQCJE7H3WWIO2OQS2MSYZFQA, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.572552794 (547h19m8.443635839s) [noise=true , address=145.79.227.7:4295 , piece_id=JVCXM5X3G75GGCJVUIAJZU5KRAP4N6UDHX5YWWFKLQ2COIQKAIGQ, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.583512634 (547h19m8.454595669s) [noise=true , address=145.79.227.48:4301 , piece_id=ZE33WA3F3VYZR6DATAHUMDX7ZBUNTJEH6AWPGDUB7CAQOTMIPS7A, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.583592996 (547h19m8.454676031s) [noise=true , address=145.79.227.42:4123 , piece_id=DAR6KC6JDNLEZWPCZ6UAOZI3THX7BPLS6LFWWOCPB73DWYHYT4TA, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.586214929 (547h19m8.457297974s) [noise=true , address=145.79.227.36:4517 , piece_id=PKXX6X2Z3R6QPS2O4V472KLCNNL3H7BFYZFFCPC6CQIQXIZUHYRQ, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.591427374 (547h19m8.462510419s) [noise=true , address=145.79.227.17:4971 , piece_id=JNM5P4N6TXORYJOKABNTFYNN5LDHDHCCRXQNCU57DQC3JGOSM4RQ, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.591979446 (547h19m8.463062501s) [noise=true , address=145.79.227.16:4910 , piece_id=K3ZYPYEWS5JMATWKIU3YPJA2M3MOTQDB3FQGTFP3SKQPR2CQE4OQ, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.611871715 (547h19m8.48295475s) [noise=true , address=145.79.227.11:4732 , piece_id=25N43O4KVYRUB2H5TQE2KODL2RJXJLW7RJCTRBCMEWWTQK35I73Q, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.829117968 (547h19m8.700201013s) [noise=true , address=45.91.116.14:4190 , piece_id=N6QHIKT4FIVGCS2RIHYSMVYXXZ4JNAW5NZ2WIBZKA7KSGPATJDKA, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.849261314 (547h19m8.720344359s) [noise=true , address=45.91.116.5:4297 , piece_id=GQ2WUQWMZB45QGWF2MIKRQSGRKJBWZEABT3Q6GO457ZWL5XHCTGQ, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.854629556 (547h19m8.725712601s) [noise=true , address=45.91.116.8:4048 , piece_id=3CU6BAIWEZA5UGH73H6ZS3MCM7P7RLDMLBCA7ZUW557ZXZ7AS5VQ, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.854649384 (547h19m8.725732419s) [seg_pos={0 0}, upload=479924]: Segment reached optimal threshold of 13 pieces. Dec 28 10:48:08.854817624 (547h19m8.725900659s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOHBYMFZQW656MI2HSQMHV5TCFXJ7FJXVNANAKJUBEO6YUD5GPZQ, seg_pos={0 0}, upload=479924]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:37336->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:37336->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37336->45.91.116.12:4228: use of closed network connection Dec 28 10:48:08.854823515 (547h19m8.725906561s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOHBYMFZQW656MI2HSQMHV5TCFXJ7FJXVNANAKJUBEO6YUD5GPZQ, seg_pos={0 0}, upload=479924]: Long tail cancellation due to optimal threshold Dec 28 10:48:08.854988700 (547h19m8.726071735s) [noise=true , address=77.90.31.11:4554 , piece_id=F3MQ42IURAHOGVIBXHSEMJBFZYRCUJ7PGEMRO6KFBUZKDHM33FXQ, seg_pos={0 0}, upload=479924]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 28 10:48:08.854994411 (547h19m8.726077446s) [noise=true , address=77.90.31.11:4554 , piece_id=F3MQ42IURAHOGVIBXHSEMJBFZYRCUJ7PGEMRO6KFBUZKDHM33FXQ, seg_pos={0 0}, upload=479924]: Long tail cancellation due to optimal threshold Dec 28 10:48:08.855145228 (547h19m8.726228283s) [noise=true , address=45.91.116.11:4119 , piece_id=S3ZWJI47R6LRWHRJY2FVQHBNKHHQNPK62KZG3ZXLV45DBRMKLLVA, seg_pos={0 0}, upload=479924]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 10:48:08.855158192 (547h19m8.726241428s) [noise=true , address=45.91.116.11:4119 , piece_id=S3ZWJI47R6LRWHRJY2FVQHBNKHHQNPK62KZG3ZXLV45DBRMKLLVA, seg_pos={0 0}, upload=479924]: Long tail cancellation due to optimal threshold Dec 28 10:48:08.855217576 (547h19m8.726300621s) [seg_pos={0 0}, upload=479924]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:48:08.855301065 (547h19m8.72638411s) [upload=479924]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:48:08.855305854 (547h19m8.726388889s) [upload=479924]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:48:09.159179451 (547h19m9.030262496s) [upload=479924]: Done uploading segments... Dec 28 10:48:09.159202525 (547h19m9.03028557s) [upload=479924]: Upload finished. err: <nil> Duration: 5.987404641s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...X..................X..............X.X.........X...XX.....] duration: 3.403637002s trace: https://jaeger.dev.storj.io/trace/65dc07c66ac2a58e FAIL (2025-12-28 10:48:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.403661318s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..............X..XX.....X..X............XX...........XXX.X.X] duration: 30.004985213s trace: https://jaeger.dev.storj.io/trace/5451b11343dd2f42 FAIL (2025-12-28 10:48:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3CU6BAIWEZA5UGH73H6ZS3MCM7P7RLDMLBCA7ZUW557ZXZ7AS5VQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: GQ2WUQWMZB45QGWF2MIKRQSGRKJBWZEABT3Q6GO457ZWL5XHCTGQ) context deadline exceeded ======LOGS====== Dec 28 10:48:06.310148263 (547h19m6.181231298s) [upload=479924]: Starting upload Dec 28 10:48:06.310369455 (547h19m6.18145249s) [upload=479924]: Done starting upload Dec 28 10:48:06.310420581 (547h19m6.181503617s) [upload=479924]: Uploading segments... Dec 28 10:48:06.310547103 (547h19m6.181630138s) [upload=479924]: Got next segment. Inline: false Dec 28 10:48:06.310576488 (547h19m6.181659533s) [upload=479924]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:48:06.310582630 (547h19m6.181665665s) [upload=479924]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:48:06.553163324 (547h19m6.424246359s) [seg_pos={0 0}, upload=479924]: Begin upload segment... Dec 28 10:48:06.553214903 (547h19m6.424297948s) [seg_pos={0 0}, upload=479924]: Done begin upload segment. Dec 28 10:48:06.553245691 (547h19m6.424328726s) [noise=true , address=145.79.227.7:4295 , piece_id=JVCXM5X3G75GGCJVUIAJZU5KRAP4N6UDHX5YWWFKLQ2COIQKAIGQ, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.553391970 (547h19m6.424475005s) [noise=true , address=77.90.31.11:4554 , piece_id=F3MQ42IURAHOGVIBXHSEMJBFZYRCUJ7PGEMRO6KFBUZKDHM33FXQ, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.553921238 (547h19m6.425004273s) [noise=true , address=145.79.227.36:4517 , piece_id=PKXX6X2Z3R6QPS2O4V472KLCNNL3H7BFYZFFCPC6CQIQXIZUHYRQ, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.554222943 (547h19m6.425305988s) [noise=true , address=45.91.116.8:4048 , piece_id=3CU6BAIWEZA5UGH73H6ZS3MCM7P7RLDMLBCA7ZUW557ZXZ7AS5VQ, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.554625168 (547h19m6.425708213s) [noise=true , address=145.79.227.46:4603 , piece_id=57GZZIEBM2NTADYLVKBICNSJZSHSUPZYHWOSO5A4NXSTW6XGZYIA, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.554914278 (547h19m6.425997313s) [noise=true , address=145.79.227.48:4301 , piece_id=ZE33WA3F3VYZR6DATAHUMDX7ZBUNTJEH6AWPGDUB7CAQOTMIPS7A, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.555198861 (547h19m6.426281926s) [noise=true , address=45.91.116.5:4297 , piece_id=GQ2WUQWMZB45QGWF2MIKRQSGRKJBWZEABT3Q6GO457ZWL5XHCTGQ, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.555539799 (547h19m6.426622834s) [noise=true , address=145.79.227.51:4410 , piece_id=OZCNZBFE4523TF3LB2B65PQZAWO4NQCJE7H3WWIO2OQS2MSYZFQA, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.555907600 (547h19m6.426990635s) [noise=true , address=145.79.227.45:4230 , piece_id=Z2VO4GCNORLXRSYL3BHUEJEYPCRGAPWOJ4XLIUJ4YCVDSYWU4ACQ, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.556957269 (547h19m6.428040314s) [noise=true , address=145.79.227.42:4123 , piece_id=DAR6KC6JDNLEZWPCZ6UAOZI3THX7BPLS6LFWWOCPB73DWYHYT4TA, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.558083172 (547h19m6.429166217s) [noise=true , address=45.91.116.11:4119 , piece_id=S3ZWJI47R6LRWHRJY2FVQHBNKHHQNPK62KZG3ZXLV45DBRMKLLVA, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.559616001 (547h19m6.430699046s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOHBYMFZQW656MI2HSQMHV5TCFXJ7FJXVNANAKJUBEO6YUD5GPZQ, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.561153680 (547h19m6.432236725s) [noise=true , address=145.79.227.16:4910 , piece_id=K3ZYPYEWS5JMATWKIU3YPJA2M3MOTQDB3FQGTFP3SKQPR2CQE4OQ, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.562976310 (547h19m6.434059345s) [noise=true , address=145.79.227.11:4732 , piece_id=25N43O4KVYRUB2H5TQE2KODL2RJXJLW7RJCTRBCMEWWTQK35I73Q, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.564075544 (547h19m6.435158579s) [noise=true , address=145.79.227.17:4971 , piece_id=JNM5P4N6TXORYJOKABNTFYNN5LDHDHCCRXQNCU57DQC3JGOSM4RQ, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:06.565284405 (547h19m6.43636744s) [noise=true , address=45.91.116.14:4190 , piece_id=N6QHIKT4FIVGCS2RIHYSMVYXXZ4JNAW5NZ2WIBZKA7KSGPATJDKA, seg_pos={0 0}, upload=479924]: Uploading piece... Dec 28 10:48:08.263029901 (547h19m8.134112936s) [upload=479924]: Next returned nil segment Dec 28 10:48:08.263056782 (547h19m8.134139817s) [upload=479924]: Waiting for error group managing segments... Dec 28 10:48:08.546196217 (547h19m8.417279252s) [noise=true , address=145.79.227.46:4603 , piece_id=57GZZIEBM2NTADYLVKBICNSJZSHSUPZYHWOSO5A4NXSTW6XGZYIA, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.560499743 (547h19m8.431582778s) [noise=true , address=145.79.227.45:4230 , piece_id=Z2VO4GCNORLXRSYL3BHUEJEYPCRGAPWOJ4XLIUJ4YCVDSYWU4ACQ, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.563591801 (547h19m8.434674846s) [noise=true , address=145.79.227.51:4410 , piece_id=OZCNZBFE4523TF3LB2B65PQZAWO4NQCJE7H3WWIO2OQS2MSYZFQA, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.572552794 (547h19m8.443635839s) [noise=true , address=145.79.227.7:4295 , piece_id=JVCXM5X3G75GGCJVUIAJZU5KRAP4N6UDHX5YWWFKLQ2COIQKAIGQ, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.583512634 (547h19m8.454595669s) [noise=true , address=145.79.227.48:4301 , piece_id=ZE33WA3F3VYZR6DATAHUMDX7ZBUNTJEH6AWPGDUB7CAQOTMIPS7A, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.583592996 (547h19m8.454676031s) [noise=true , address=145.79.227.42:4123 , piece_id=DAR6KC6JDNLEZWPCZ6UAOZI3THX7BPLS6LFWWOCPB73DWYHYT4TA, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.586214929 (547h19m8.457297974s) [noise=true , address=145.79.227.36:4517 , piece_id=PKXX6X2Z3R6QPS2O4V472KLCNNL3H7BFYZFFCPC6CQIQXIZUHYRQ, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.591427374 (547h19m8.462510419s) [noise=true , address=145.79.227.17:4971 , piece_id=JNM5P4N6TXORYJOKABNTFYNN5LDHDHCCRXQNCU57DQC3JGOSM4RQ, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.591979446 (547h19m8.463062501s) [noise=true , address=145.79.227.16:4910 , piece_id=K3ZYPYEWS5JMATWKIU3YPJA2M3MOTQDB3FQGTFP3SKQPR2CQE4OQ, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.611871715 (547h19m8.48295475s) [noise=true , address=145.79.227.11:4732 , piece_id=25N43O4KVYRUB2H5TQE2KODL2RJXJLW7RJCTRBCMEWWTQK35I73Q, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.829117968 (547h19m8.700201013s) [noise=true , address=45.91.116.14:4190 , piece_id=N6QHIKT4FIVGCS2RIHYSMVYXXZ4JNAW5NZ2WIBZKA7KSGPATJDKA, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.849261314 (547h19m8.720344359s) [noise=true , address=45.91.116.5:4297 , piece_id=GQ2WUQWMZB45QGWF2MIKRQSGRKJBWZEABT3Q6GO457ZWL5XHCTGQ, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.854629556 (547h19m8.725712601s) [noise=true , address=45.91.116.8:4048 , piece_id=3CU6BAIWEZA5UGH73H6ZS3MCM7P7RLDMLBCA7ZUW557ZXZ7AS5VQ, seg_pos={0 0}, upload=479924]: Done uploading piece. err: <nil> Dec 28 10:48:08.854649384 (547h19m8.725732419s) [seg_pos={0 0}, upload=479924]: Segment reached optimal threshold of 13 pieces. Dec 28 10:48:08.854817624 (547h19m8.725900659s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOHBYMFZQW656MI2HSQMHV5TCFXJ7FJXVNANAKJUBEO6YUD5GPZQ, seg_pos={0 0}, upload=479924]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:37336->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:37336->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37336->45.91.116.12:4228: use of closed network connection Dec 28 10:48:08.854823515 (547h19m8.725906561s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOHBYMFZQW656MI2HSQMHV5TCFXJ7FJXVNANAKJUBEO6YUD5GPZQ, seg_pos={0 0}, upload=479924]: Long tail cancellation due to optimal threshold Dec 28 10:48:08.854988700 (547h19m8.726071735s) [noise=true , address=77.90.31.11:4554 , piece_id=F3MQ42IURAHOGVIBXHSEMJBFZYRCUJ7PGEMRO6KFBUZKDHM33FXQ, seg_pos={0 0}, upload=479924]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 28 10:48:08.854994411 (547h19m8.726077446s) [noise=true , address=77.90.31.11:4554 , piece_id=F3MQ42IURAHOGVIBXHSEMJBFZYRCUJ7PGEMRO6KFBUZKDHM33FXQ, seg_pos={0 0}, upload=479924]: Long tail cancellation due to optimal threshold Dec 28 10:48:08.855145228 (547h19m8.726228283s) [noise=true , address=45.91.116.11:4119 , piece_id=S3ZWJI47R6LRWHRJY2FVQHBNKHHQNPK62KZG3ZXLV45DBRMKLLVA, seg_pos={0 0}, upload=479924]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 10:48:08.855158192 (547h19m8.726241428s) [noise=true , address=45.91.116.11:4119 , piece_id=S3ZWJI47R6LRWHRJY2FVQHBNKHHQNPK62KZG3ZXLV45DBRMKLLVA, seg_pos={0 0}, upload=479924]: Long tail cancellation due to optimal threshold Dec 28 10:48:08.855217576 (547h19m8.726300621s) [seg_pos={0 0}, upload=479924]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:48:08.855301065 (547h19m8.72638411s) [upload=479924]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:48:08.855305854 (547h19m8.726388889s) [upload=479924]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:48:09.159179451 (547h19m9.030262496s) [upload=479924]: Done uploading segments... Dec 28 10:48:09.159202525 (547h19m9.03028557s) [upload=479924]: Upload finished. err: <nil> Duration: 30.005006324s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....X..................X..............X.X.........X...XX....] duration: 3.509837685s trace: https://jaeger.dev.storj.io/trace/65dc07c66ac2a58e FAIL (2025-12-28 10:48:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.509864967s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....X..................X..............X.X.........X...XX...] duration: 3.620918451s trace: https://jaeger.dev.storj.io/trace/65dc07c66ac2a58e FAIL (2025-12-28 10:47:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.620948399s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...............X..XX.....X..X............XX...........XXX.X.] duration: 5.720467398s trace: https://jaeger.dev.storj.io/trace/224eb0cb0d4cfaa2 FAIL (2025-12-28 10:46:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: KQWZSUUTQ5OVGGMKAZCUPNXCPR7HFA444SSD2YZFVTWMUXAMT3IQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: UDFQKADWO3KYCIKO6FUJ5OMYFM45HTH2ZV5VBWYYEDQPXLA4DWVA) context deadline exceeded ======LOGS====== Dec 28 10:45:42.805378505 (547h16m42.67646154s) [upload=479889]: Starting upload Dec 28 10:45:42.805434071 (547h16m42.676517106s) [upload=479889]: Done starting upload Dec 28 10:45:42.805521608 (547h16m42.676604653s) [upload=479889]: Uploading segments... Dec 28 10:45:42.805605487 (547h16m42.676688552s) [upload=479889]: Got next segment. Inline: false Dec 28 10:45:42.805675691 (547h16m42.676758726s) [upload=479889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:45:42.805683717 (547h16m42.676766772s) [upload=479889]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:45:43.051240358 (547h16m42.922323393s) [seg_pos={0 0}, upload=479889]: Begin upload segment... Dec 28 10:45:43.051292277 (547h16m42.922375312s) [seg_pos={0 0}, upload=479889]: Done begin upload segment. Dec 28 10:45:43.051323035 (547h16m42.92240607s) [noise=true , address=145.79.227.23:4547 , piece_id=H6QPS2B7C4ZM3W4KX67I5KMQLKLDVAFGYKL5W7RYE3ONJCPXV6TQ, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.051453784 (547h16m42.922536829s) [noise=true , address=145.79.227.10:4401 , piece_id=ENRAGJEGQVSBAIIQGLURVZHPX6EI7UOBO5KYA5YL5QXA5VUDGL3A, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.051989395 (547h16m42.92307244s) [noise=true , address=145.79.227.11:4730 , piece_id=4AZPT6UJPK6X5LBM675UBK5T26KQK7RWEDTUUY2HXHEM2DE5PYZQ, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.052315576 (547h16m42.923398621s) [noise=true , address=145.79.227.15:4875 , piece_id=MVMKSVUWHO24HEFMNLF6LDUCHOU3ECXIKH2BUEQMG42DT4AFQHMA, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.052495067 (547h16m42.923578102s) [noise=true , address=145.79.227.7:4294 , piece_id=HLTE3ZQ4L3HQTR45X3W2FQVFGWCFK2T5IQ3OSZ2Q7IE2X5UC4BRQ, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.052879679 (547h16m42.923962725s) [noise=true , address=145.79.227.41:4087 , piece_id=7CCMAJXD5S47OJR5H6HX3PAWFN6X5MKSLWBBHDJDGYJ6PPIPIQVQ, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.052908374 (547h16m42.923991419s) [noise=true , address=45.91.116.4:4937 , piece_id=2355JR54HLSQNEVFAPX6OE27UFCFKVIVBWFLE2DB6GN3QJUBZ5VA, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.053236689 (547h16m42.924319724s) [noise=true , address=77.90.31.13:4627 , piece_id=UDFQKADWO3KYCIKO6FUJ5OMYFM45HTH2ZV5VBWYYEDQPXLA4DWVA, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.053367959 (547h16m42.924451004s) [noise=true , address=145.79.227.39:4013 , piece_id=KLVDXICS4TMUQGQ2TL22S6JYJQZPFHPNXBZD3QZ2JMBZF7ZUFJJA, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.053507515 (547h16m42.92459055s) [noise=true , address=145.79.227.6:4259 , piece_id=6LYWXYSXIHB2JN7OLMC326EK53L6Q7DFITBHJI6VFRKMGS7LGATQ, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.053766108 (547h16m42.924849143s) [noise=true , address=145.79.227.17:4972 , piece_id=JFMXQ6DA4NA5KZAKMJWDAAIX2EHHODXYJBTXZP2ZZV2VEH54CUZQ, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.053859725 (547h16m42.924942761s) [noise=true , address=45.91.116.9:4261 , piece_id=KQWZSUUTQ5OVGGMKAZCUPNXCPR7HFA444SSD2YZFVTWMUXAMT3IQ, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.054218810 (547h16m42.925301845s) [noise=true , address=45.91.116.7:4010 , piece_id=B3NLI3MCBGKNS7STPV7XKV2UMZVO3PQ7NOSZ5KBSVISMUPDBWYLQ, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.055376694 (547h16m42.92645973s) [noise=true , address=145.79.227.49:4339 , piece_id=SWVV6S7RXIRQNS4PEOQGBHU2IQNCZUU6ZRLIOXZCP4Y4B524R6XQ, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.056219199 (547h16m42.927302234s) [noise=true , address=45.91.116.8:4047 , piece_id=BRSV3DAPABV3AYCPIYE35UJKC2WYPYQYISD3EJHKIWL4TCDQIE2Q, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.057513844 (547h16m42.928597149s) [noise=true , address=145.79.227.40:4050 , piece_id=RZBS5PL6CNP4EHT4WCHMNCYLKA6ABUPHFAUMNR4LTMIIVEVX3HMA, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:44.675514320 (547h16m44.546597355s) [upload=479889]: Next returned nil segment Dec 28 10:45:44.675527806 (547h16m44.546610841s) [upload=479889]: Waiting for error group managing segments... Dec 28 10:45:45.061586357 (547h16m44.932669392s) [noise=true , address=145.79.227.15:4875 , piece_id=MVMKSVUWHO24HEFMNLF6LDUCHOU3ECXIKH2BUEQMG42DT4AFQHMA, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.066774216 (547h16m44.937857251s) [noise=true , address=145.79.227.23:4547 , piece_id=H6QPS2B7C4ZM3W4KX67I5KMQLKLDVAFGYKL5W7RYE3ONJCPXV6TQ, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.067264549 (547h16m44.938347584s) [noise=true , address=145.79.227.17:4972 , piece_id=JFMXQ6DA4NA5KZAKMJWDAAIX2EHHODXYJBTXZP2ZZV2VEH54CUZQ, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.072294678 (547h16m44.943377713s) [noise=true , address=145.79.227.39:4013 , piece_id=KLVDXICS4TMUQGQ2TL22S6JYJQZPFHPNXBZD3QZ2JMBZF7ZUFJJA, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.072828444 (547h16m44.943911479s) [noise=true , address=145.79.227.6:4259 , piece_id=6LYWXYSXIHB2JN7OLMC326EK53L6Q7DFITBHJI6VFRKMGS7LGATQ, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.073183911 (547h16m44.944266956s) [noise=true , address=145.79.227.11:4730 , piece_id=4AZPT6UJPK6X5LBM675UBK5T26KQK7RWEDTUUY2HXHEM2DE5PYZQ, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.074909707 (547h16m44.945992742s) [noise=true , address=145.79.227.41:4087 , piece_id=7CCMAJXD5S47OJR5H6HX3PAWFN6X5MKSLWBBHDJDGYJ6PPIPIQVQ, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.084088885 (547h16m44.95517192s) [noise=true , address=145.79.227.10:4401 , piece_id=ENRAGJEGQVSBAIIQGLURVZHPX6EI7UOBO5KYA5YL5QXA5VUDGL3A, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.094189570 (547h16m44.965272605s) [noise=true , address=145.79.227.7:4294 , piece_id=HLTE3ZQ4L3HQTR45X3W2FQVFGWCFK2T5IQ3OSZ2Q7IE2X5UC4BRQ, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.233717995 (547h16m45.10480104s) [noise=true , address=77.90.31.13:4627 , piece_id=UDFQKADWO3KYCIKO6FUJ5OMYFM45HTH2ZV5VBWYYEDQPXLA4DWVA, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.298523745 (547h16m45.16960678s) [noise=true , address=45.91.116.4:4937 , piece_id=2355JR54HLSQNEVFAPX6OE27UFCFKVIVBWFLE2DB6GN3QJUBZ5VA, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.375919932 (547h16m45.247002967s) [noise=true , address=45.91.116.8:4047 , piece_id=BRSV3DAPABV3AYCPIYE35UJKC2WYPYQYISD3EJHKIWL4TCDQIE2Q, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.418792929 (547h16m45.289875974s) [noise=true , address=45.91.116.9:4261 , piece_id=KQWZSUUTQ5OVGGMKAZCUPNXCPR7HFA444SSD2YZFVTWMUXAMT3IQ, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.418868162 (547h16m45.289951227s) [seg_pos={0 0}, upload=479889]: Segment reached optimal threshold of 13 pieces. Dec 28 10:45:45.419145690 (547h16m45.290228725s) [noise=true , address=145.79.227.40:4050 , piece_id=RZBS5PL6CNP4EHT4WCHMNCYLKA6ABUPHFAUMNR4LTMIIVEVX3HMA, seg_pos={0 0}, upload=479889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 28 10:45:45.419154206 (547h16m45.290237241s) [noise=true , address=145.79.227.40:4050 , piece_id=RZBS5PL6CNP4EHT4WCHMNCYLKA6ABUPHFAUMNR4LTMIIVEVX3HMA, seg_pos={0 0}, upload=479889]: Long tail cancellation due to optimal threshold Dec 28 10:45:45.419281940 (547h16m45.290364975s) [noise=true , address=45.91.116.7:4010 , piece_id=B3NLI3MCBGKNS7STPV7XKV2UMZVO3PQ7NOSZ5KBSVISMUPDBWYLQ, seg_pos={0 0}, upload=479889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:50012->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:50012->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50012->45.91.116.7:4010: use of closed network connection Dec 28 10:45:45.419300065 (547h16m45.2903831s) [noise=true , address=45.91.116.7:4010 , piece_id=B3NLI3MCBGKNS7STPV7XKV2UMZVO3PQ7NOSZ5KBSVISMUPDBWYLQ, seg_pos={0 0}, upload=479889]: Long tail cancellation due to optimal threshold Dec 28 10:45:45.419366622 (547h16m45.290449657s) [noise=true , address=145.79.227.49:4339 , piece_id=SWVV6S7RXIRQNS4PEOQGBHU2IQNCZUU6ZRLIOXZCP4Y4B524R6XQ, seg_pos={0 0}, upload=479889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 28 10:45:45.419373354 (547h16m45.290456399s) [noise=true , address=145.79.227.49:4339 , piece_id=SWVV6S7RXIRQNS4PEOQGBHU2IQNCZUU6ZRLIOXZCP4Y4B524R6XQ, seg_pos={0 0}, upload=479889]: Long tail cancellation due to optimal threshold Dec 28 10:45:45.419479777 (547h16m45.290562812s) [seg_pos={0 0}, upload=479889]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:45:45.419630544 (547h16m45.290713579s) [upload=479889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:45:45.419637356 (547h16m45.290720391s) [upload=479889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:45:45.702440852 (547h16m45.573523887s) [upload=479889]: Done uploading segments... Dec 28 10:45:45.702463816 (547h16m45.573546851s) [upload=479889]: Upload finished. err: <nil> Duration: 5.720497196s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [X.....X..................X..............X.X.........X...XX..] duration: 4.051661502s trace: https://jaeger.dev.storj.io/trace/65dc07c66ac2a58e FAIL (2025-12-28 10:46:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.051691699s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [................X..XX.....X..X............XX...........XXX.X] duration: 30.003602941s trace: https://jaeger.dev.storj.io/trace/224eb0cb0d4cfaa2 FAIL (2025-12-28 10:46:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: KQWZSUUTQ5OVGGMKAZCUPNXCPR7HFA444SSD2YZFVTWMUXAMT3IQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: UDFQKADWO3KYCIKO6FUJ5OMYFM45HTH2ZV5VBWYYEDQPXLA4DWVA) context deadline exceeded ======LOGS====== Dec 28 10:45:42.805378505 (547h16m42.67646154s) [upload=479889]: Starting upload Dec 28 10:45:42.805434071 (547h16m42.676517106s) [upload=479889]: Done starting upload Dec 28 10:45:42.805521608 (547h16m42.676604653s) [upload=479889]: Uploading segments... Dec 28 10:45:42.805605487 (547h16m42.676688552s) [upload=479889]: Got next segment. Inline: false Dec 28 10:45:42.805675691 (547h16m42.676758726s) [upload=479889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:45:42.805683717 (547h16m42.676766772s) [upload=479889]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:45:43.051240358 (547h16m42.922323393s) [seg_pos={0 0}, upload=479889]: Begin upload segment... Dec 28 10:45:43.051292277 (547h16m42.922375312s) [seg_pos={0 0}, upload=479889]: Done begin upload segment. Dec 28 10:45:43.051323035 (547h16m42.92240607s) [noise=true , address=145.79.227.23:4547 , piece_id=H6QPS2B7C4ZM3W4KX67I5KMQLKLDVAFGYKL5W7RYE3ONJCPXV6TQ, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.051453784 (547h16m42.922536829s) [noise=true , address=145.79.227.10:4401 , piece_id=ENRAGJEGQVSBAIIQGLURVZHPX6EI7UOBO5KYA5YL5QXA5VUDGL3A, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.051989395 (547h16m42.92307244s) [noise=true , address=145.79.227.11:4730 , piece_id=4AZPT6UJPK6X5LBM675UBK5T26KQK7RWEDTUUY2HXHEM2DE5PYZQ, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.052315576 (547h16m42.923398621s) [noise=true , address=145.79.227.15:4875 , piece_id=MVMKSVUWHO24HEFMNLF6LDUCHOU3ECXIKH2BUEQMG42DT4AFQHMA, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.052495067 (547h16m42.923578102s) [noise=true , address=145.79.227.7:4294 , piece_id=HLTE3ZQ4L3HQTR45X3W2FQVFGWCFK2T5IQ3OSZ2Q7IE2X5UC4BRQ, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.052879679 (547h16m42.923962725s) [noise=true , address=145.79.227.41:4087 , piece_id=7CCMAJXD5S47OJR5H6HX3PAWFN6X5MKSLWBBHDJDGYJ6PPIPIQVQ, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.052908374 (547h16m42.923991419s) [noise=true , address=45.91.116.4:4937 , piece_id=2355JR54HLSQNEVFAPX6OE27UFCFKVIVBWFLE2DB6GN3QJUBZ5VA, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.053236689 (547h16m42.924319724s) [noise=true , address=77.90.31.13:4627 , piece_id=UDFQKADWO3KYCIKO6FUJ5OMYFM45HTH2ZV5VBWYYEDQPXLA4DWVA, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.053367959 (547h16m42.924451004s) [noise=true , address=145.79.227.39:4013 , piece_id=KLVDXICS4TMUQGQ2TL22S6JYJQZPFHPNXBZD3QZ2JMBZF7ZUFJJA, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.053507515 (547h16m42.92459055s) [noise=true , address=145.79.227.6:4259 , piece_id=6LYWXYSXIHB2JN7OLMC326EK53L6Q7DFITBHJI6VFRKMGS7LGATQ, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.053766108 (547h16m42.924849143s) [noise=true , address=145.79.227.17:4972 , piece_id=JFMXQ6DA4NA5KZAKMJWDAAIX2EHHODXYJBTXZP2ZZV2VEH54CUZQ, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.053859725 (547h16m42.924942761s) [noise=true , address=45.91.116.9:4261 , piece_id=KQWZSUUTQ5OVGGMKAZCUPNXCPR7HFA444SSD2YZFVTWMUXAMT3IQ, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.054218810 (547h16m42.925301845s) [noise=true , address=45.91.116.7:4010 , piece_id=B3NLI3MCBGKNS7STPV7XKV2UMZVO3PQ7NOSZ5KBSVISMUPDBWYLQ, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.055376694 (547h16m42.92645973s) [noise=true , address=145.79.227.49:4339 , piece_id=SWVV6S7RXIRQNS4PEOQGBHU2IQNCZUU6ZRLIOXZCP4Y4B524R6XQ, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.056219199 (547h16m42.927302234s) [noise=true , address=45.91.116.8:4047 , piece_id=BRSV3DAPABV3AYCPIYE35UJKC2WYPYQYISD3EJHKIWL4TCDQIE2Q, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:43.057513844 (547h16m42.928597149s) [noise=true , address=145.79.227.40:4050 , piece_id=RZBS5PL6CNP4EHT4WCHMNCYLKA6ABUPHFAUMNR4LTMIIVEVX3HMA, seg_pos={0 0}, upload=479889]: Uploading piece... Dec 28 10:45:44.675514320 (547h16m44.546597355s) [upload=479889]: Next returned nil segment Dec 28 10:45:44.675527806 (547h16m44.546610841s) [upload=479889]: Waiting for error group managing segments... Dec 28 10:45:45.061586357 (547h16m44.932669392s) [noise=true , address=145.79.227.15:4875 , piece_id=MVMKSVUWHO24HEFMNLF6LDUCHOU3ECXIKH2BUEQMG42DT4AFQHMA, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.066774216 (547h16m44.937857251s) [noise=true , address=145.79.227.23:4547 , piece_id=H6QPS2B7C4ZM3W4KX67I5KMQLKLDVAFGYKL5W7RYE3ONJCPXV6TQ, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.067264549 (547h16m44.938347584s) [noise=true , address=145.79.227.17:4972 , piece_id=JFMXQ6DA4NA5KZAKMJWDAAIX2EHHODXYJBTXZP2ZZV2VEH54CUZQ, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.072294678 (547h16m44.943377713s) [noise=true , address=145.79.227.39:4013 , piece_id=KLVDXICS4TMUQGQ2TL22S6JYJQZPFHPNXBZD3QZ2JMBZF7ZUFJJA, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.072828444 (547h16m44.943911479s) [noise=true , address=145.79.227.6:4259 , piece_id=6LYWXYSXIHB2JN7OLMC326EK53L6Q7DFITBHJI6VFRKMGS7LGATQ, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.073183911 (547h16m44.944266956s) [noise=true , address=145.79.227.11:4730 , piece_id=4AZPT6UJPK6X5LBM675UBK5T26KQK7RWEDTUUY2HXHEM2DE5PYZQ, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.074909707 (547h16m44.945992742s) [noise=true , address=145.79.227.41:4087 , piece_id=7CCMAJXD5S47OJR5H6HX3PAWFN6X5MKSLWBBHDJDGYJ6PPIPIQVQ, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.084088885 (547h16m44.95517192s) [noise=true , address=145.79.227.10:4401 , piece_id=ENRAGJEGQVSBAIIQGLURVZHPX6EI7UOBO5KYA5YL5QXA5VUDGL3A, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.094189570 (547h16m44.965272605s) [noise=true , address=145.79.227.7:4294 , piece_id=HLTE3ZQ4L3HQTR45X3W2FQVFGWCFK2T5IQ3OSZ2Q7IE2X5UC4BRQ, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.233717995 (547h16m45.10480104s) [noise=true , address=77.90.31.13:4627 , piece_id=UDFQKADWO3KYCIKO6FUJ5OMYFM45HTH2ZV5VBWYYEDQPXLA4DWVA, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.298523745 (547h16m45.16960678s) [noise=true , address=45.91.116.4:4937 , piece_id=2355JR54HLSQNEVFAPX6OE27UFCFKVIVBWFLE2DB6GN3QJUBZ5VA, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.375919932 (547h16m45.247002967s) [noise=true , address=45.91.116.8:4047 , piece_id=BRSV3DAPABV3AYCPIYE35UJKC2WYPYQYISD3EJHKIWL4TCDQIE2Q, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.418792929 (547h16m45.289875974s) [noise=true , address=45.91.116.9:4261 , piece_id=KQWZSUUTQ5OVGGMKAZCUPNXCPR7HFA444SSD2YZFVTWMUXAMT3IQ, seg_pos={0 0}, upload=479889]: Done uploading piece. err: <nil> Dec 28 10:45:45.418868162 (547h16m45.289951227s) [seg_pos={0 0}, upload=479889]: Segment reached optimal threshold of 13 pieces. Dec 28 10:45:45.419145690 (547h16m45.290228725s) [noise=true , address=145.79.227.40:4050 , piece_id=RZBS5PL6CNP4EHT4WCHMNCYLKA6ABUPHFAUMNR4LTMIIVEVX3HMA, seg_pos={0 0}, upload=479889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 28 10:45:45.419154206 (547h16m45.290237241s) [noise=true , address=145.79.227.40:4050 , piece_id=RZBS5PL6CNP4EHT4WCHMNCYLKA6ABUPHFAUMNR4LTMIIVEVX3HMA, seg_pos={0 0}, upload=479889]: Long tail cancellation due to optimal threshold Dec 28 10:45:45.419281940 (547h16m45.290364975s) [noise=true , address=45.91.116.7:4010 , piece_id=B3NLI3MCBGKNS7STPV7XKV2UMZVO3PQ7NOSZ5KBSVISMUPDBWYLQ, seg_pos={0 0}, upload=479889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:50012->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:50012->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50012->45.91.116.7:4010: use of closed network connection Dec 28 10:45:45.419300065 (547h16m45.2903831s) [noise=true , address=45.91.116.7:4010 , piece_id=B3NLI3MCBGKNS7STPV7XKV2UMZVO3PQ7NOSZ5KBSVISMUPDBWYLQ, seg_pos={0 0}, upload=479889]: Long tail cancellation due to optimal threshold Dec 28 10:45:45.419366622 (547h16m45.290449657s) [noise=true , address=145.79.227.49:4339 , piece_id=SWVV6S7RXIRQNS4PEOQGBHU2IQNCZUU6ZRLIOXZCP4Y4B524R6XQ, seg_pos={0 0}, upload=479889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 28 10:45:45.419373354 (547h16m45.290456399s) [noise=true , address=145.79.227.49:4339 , piece_id=SWVV6S7RXIRQNS4PEOQGBHU2IQNCZUU6ZRLIOXZCP4Y4B524R6XQ, seg_pos={0 0}, upload=479889]: Long tail cancellation due to optimal threshold Dec 28 10:45:45.419479777 (547h16m45.290562812s) [seg_pos={0 0}, upload=479889]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:45:45.419630544 (547h16m45.290713579s) [upload=479889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:45:45.419637356 (547h16m45.290720391s) [upload=479889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:45:45.702440852 (547h16m45.573523887s) [upload=479889]: Done uploading segments... Dec 28 10:45:45.702463816 (547h16m45.573546851s) [upload=479889]: Upload finished. err: <nil> Duration: 30.003630411s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.X.....X..................X..............X.X.........X...XX.] duration: 4.515715908s trace: https://jaeger.dev.storj.io/trace/65dc07c66ac2a58e FAIL (2025-12-28 10:45:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.515764891s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..X.....X..................X..............X.X.........X...XX] duration: 5.001259032s trace: https://jaeger.dev.storj.io/trace/65dc07c66ac2a58e FAIL (2025-12-28 10:44:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001271536s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.................X..XX.....X..X............XX...........XXX.] duration: 6.13533152s trace: https://jaeger.dev.storj.io/trace/62e837a257ebb71e FAIL (2025-12-28 10:44:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: 6T2UO2ZR4KMKWCVFWLXLQBZZHVPLA7NZ5Q5FE2LA4D4QWIMONV5Q) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: EI5HQVD5NURJ7NIFP22ORTDMD4I2NUFYJRWUEIPR4LPAY2SSRAGQ) context deadline exceeded ======LOGS====== ing segments... Dec 28 10:43:47.993729977 (547h14m47.864813022s) [upload=479861]: Got next segment. Inline: false Dec 28 10:43:47.993758972 (547h14m47.864842017s) [upload=479861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:43:47.993763801 (547h14m47.864846856s) [upload=479861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:43:48.239510154 (547h14m48.110593199s) [seg_pos={0 0}, upload=479861]: Begin upload segment... Dec 28 10:43:48.239638047 (547h14m48.110721082s) [seg_pos={0 0}, upload=479861]: Done begin upload segment. Dec 28 10:43:48.239680799 (547h14m48.110763834s) [noise=true , address=145.79.227.13:4802 , piece_id=UPP4RBDB57UTJNTRCZE35WZRCNHED7KWOQGHSKAIS4BGIO45W2WA, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.239701478 (547h14m48.110784523s) [noise=true , address=145.79.227.28:4725 , piece_id=LIUFRUL3HHHJD7LGBALSB4T4EONJOXH22ZIE2EALST7JTW6U2P2Q, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.240279609 (547h14m48.111362644s) [noise=true , address=145.79.227.51:4411 , piece_id=LEGCPWKSOQCINJPJTU5MPL5NKXUNUATLHN2LKOHEEI6UTOZOFRZQ, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.240519035 (547h14m48.11160207s) [noise=true , address=145.79.227.38:4978 , piece_id=PXQN7MTB72NZLJFNR5S46U2CSH7AGD6YIUYTQ7E7HBXWNUHMJEXQ, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.240913406 (547h14m48.111996441s) [noise=true , address=77.90.31.11:4553 , piece_id=NRWBWZCPDVJRSIRSAF46BWQ3JLYHDGYE5LZLDCNYU4XYLYYOBMJQ, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.241221763 (547h14m48.112304798s) [noise=true , address=145.79.227.43:4159 , piece_id=GQUP7QOG2HDMVII54GZI6XJUHCSH3VT3FQEKQYTDNO2HTAXRZHVQ, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.241556029 (547h14m48.112639064s) [noise=true , address=45.91.116.9:4261 , piece_id=W5YWJ26CS62BZR6XB7MLNA7MNCBWMXLMIKTTPQJHT2VB5BRXQOHQ, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.241573172 (547h14m48.112656217s) [noise=true , address=45.91.116.11:4120 , piece_id=KWNGXR6YOP7JSYTS2GPKJGAZTOEVRERX4BEPIYH5QUZSFLIHRGBA, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.241846373 (547h14m48.112929408s) [noise=true , address=45.91.116.6:4974 , piece_id=NNLBTFSPU3EKPZT7MQYHY5NDGHVNJWNNUP7NVEZSNDKLVDYN6XTA, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.242169838 (547h14m48.113252883s) [noise=true , address=45.91.116.4:4940 , piece_id=2UNFUCWWVNFGMJOXNPYEPSCOUCMHRE5BGVLWJVUEGZF6O6WKIJLQ, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.242828072 (547h14m48.113911117s) [noise=true , address=145.79.227.40:4051 , piece_id=HO7ZKUXK5IANQJ2RKG73CSCUVQBU4S3WFRGPOG7NTZUMF3RX7CHA, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.243389651 (547h14m48.114472706s) [noise=true , address=145.79.227.18:4007 , piece_id=55SIKCGSFOBUOFYQUW4QKIRKYFUGNFOQROEX4EWUHSRUL2CY4DIA, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.243854347 (547h14m48.114937402s) [noise=true , address=145.79.227.15:4876 , piece_id=IYSDJFCGLMLZHTRMMQGQHS3ZGESSKGP3KG4ZO4UIP65C6KOTITGQ, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.243959747 (547h14m48.115042792s) [noise=true , address=77.90.31.14:4662 , piece_id=EI5HQVD5NURJ7NIFP22ORTDMD4I2NUFYJRWUEIPR4LPAY2SSRAGQ, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.244477753 (547h14m48.115560798s) [noise=true , address=77.90.31.21:4878 , piece_id=6T2UO2ZR4KMKWCVFWLXLQBZZHVPLA7NZ5Q5FE2LA4D4QWIMONV5Q, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.244515265 (547h14m48.1155983s) [noise=true , address=145.79.227.14:4837 , piece_id=LM6GUXOST2SE36356FAVFCPRJEYOKNGLVNNT4Q7QQBERMJR4AQFA, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:49.866345704 (547h14m49.737428739s) [upload=479861]: Next returned nil segment Dec 28 10:43:49.866398145 (547h14m49.73748118s) [upload=479861]: Waiting for error group managing segments... Dec 28 10:43:50.248239055 (547h14m50.1193221s) [noise=true , address=145.79.227.40:4051 , piece_id=HO7ZKUXK5IANQJ2RKG73CSCUVQBU4S3WFRGPOG7NTZUMF3RX7CHA, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.249951525 (547h14m50.12103457s) [noise=true , address=145.79.227.28:4725 , piece_id=LIUFRUL3HHHJD7LGBALSB4T4EONJOXH22ZIE2EALST7JTW6U2P2Q, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.253414459 (547h14m50.124497504s) [noise=true , address=145.79.227.14:4837 , piece_id=LM6GUXOST2SE36356FAVFCPRJEYOKNGLVNNT4Q7QQBERMJR4AQFA, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.255672138 (547h14m50.126755183s) [noise=true , address=145.79.227.43:4159 , piece_id=GQUP7QOG2HDMVII54GZI6XJUHCSH3VT3FQEKQYTDNO2HTAXRZHVQ, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.257090669 (547h14m50.128173704s) [noise=true , address=145.79.227.18:4007 , piece_id=55SIKCGSFOBUOFYQUW4QKIRKYFUGNFOQROEX4EWUHSRUL2CY4DIA, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.263424249 (547h14m50.134507284s) [noise=true , address=145.79.227.51:4411 , piece_id=LEGCPWKSOQCINJPJTU5MPL5NKXUNUATLHN2LKOHEEI6UTOZOFRZQ, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.266553507 (547h14m50.137636542s) [noise=true , address=145.79.227.15:4876 , piece_id=IYSDJFCGLMLZHTRMMQGQHS3ZGESSKGP3KG4ZO4UIP65C6KOTITGQ, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.270217174 (547h14m50.141300219s) [noise=true , address=145.79.227.13:4802 , piece_id=UPP4RBDB57UTJNTRCZE35WZRCNHED7KWOQGHSKAIS4BGIO45W2WA, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.270691116 (547h14m50.141774161s) [noise=true , address=145.79.227.38:4978 , piece_id=PXQN7MTB72NZLJFNR5S46U2CSH7AGD6YIUYTQ7E7HBXWNUHMJEXQ, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.462796297 (547h14m50.333879342s) [noise=true , address=77.90.31.14:4662 , piece_id=EI5HQVD5NURJ7NIFP22ORTDMD4I2NUFYJRWUEIPR4LPAY2SSRAGQ, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.550645136 (547h14m50.421728171s) [noise=true , address=77.90.31.21:4878 , piece_id=6T2UO2ZR4KMKWCVFWLXLQBZZHVPLA7NZ5Q5FE2LA4D4QWIMONV5Q, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.642176397 (547h14m50.513259432s) [noise=true , address=45.91.116.11:4120 , piece_id=KWNGXR6YOP7JSYTS2GPKJGAZTOEVRERX4BEPIYH5QUZSFLIHRGBA, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.660209867 (547h14m50.531292902s) [noise=true , address=45.91.116.9:4261 , piece_id=W5YWJ26CS62BZR6XB7MLNA7MNCBWMXLMIKTTPQJHT2VB5BRXQOHQ, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.660237849 (547h14m50.531320894s) [seg_pos={0 0}, upload=479861]: Segment reached optimal threshold of 13 pieces. Dec 28 10:43:50.660588547 (547h14m50.531671602s) [noise=true , address=45.91.116.4:4940 , piece_id=2UNFUCWWVNFGMJOXNPYEPSCOUCMHRE5BGVLWJVUEGZF6O6WKIJLQ, seg_pos={0 0}, upload=479861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 28 10:43:50.660599238 (547h14m50.531682283s) [noise=true , address=45.91.116.4:4940 , piece_id=2UNFUCWWVNFGMJOXNPYEPSCOUCMHRE5BGVLWJVUEGZF6O6WKIJLQ, seg_pos={0 0}, upload=479861]: Long tail cancellation due to optimal threshold Dec 28 10:43:50.660615238 (547h14m50.531698273s) [noise=true , address=77.90.31.11:4553 , piece_id=NRWBWZCPDVJRSIRSAF46BWQ3JLYHDGYE5LZLDCNYU4XYLYYOBMJQ, seg_pos={0 0}, upload=479861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:54056->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:54056->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54056->77.90.31.11:4553: use of closed network connection Dec 28 10:43:50.660644093 (547h14m50.531727138s) [noise=true , address=77.90.31.11:4553 , piece_id=NRWBWZCPDVJRSIRSAF46BWQ3JLYHDGYE5LZLDCNYU4XYLYYOBMJQ, seg_pos={0 0}, upload=479861]: Long tail cancellation due to optimal threshold Dec 28 10:43:50.660865855 (547h14m50.53194889s) [noise=true , address=45.91.116.6:4974 , piece_id=NNLBTFSPU3EKPZT7MQYHY5NDGHVNJWNNUP7NVEZSNDKLVDYN6XTA, seg_pos={0 0}, upload=479861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:46496->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:46496->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46496->45.91.116.6:4974: use of closed network connection Dec 28 10:43:50.660889400 (547h14m50.531972435s) [noise=true , address=45.91.116.6:4974 , piece_id=NNLBTFSPU3EKPZT7MQYHY5NDGHVNJWNNUP7NVEZSNDKLVDYN6XTA, seg_pos={0 0}, upload=479861]: Long tail cancellation due to optimal threshold Dec 28 10:43:50.660908677 (547h14m50.531991722s) [seg_pos={0 0}, upload=479861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:43:50.661021372 (547h14m50.532104407s) [upload=479861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:43:50.661026040 (547h14m50.532109085s) [upload=479861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:43:50.960995142 (547h14m50.832078217s) [upload=479861]: Done uploading segments... Dec 28 10:43:50.961021773 (547h14m50.832104828s) [upload=479861]: Upload finished. err: <nil> Duration: 6.135356819s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................X..XX.....X..X............XX...........XXX] duration: 30.003176399s trace: https://jaeger.dev.storj.io/trace/62e837a257ebb71e FAIL (2025-12-28 10:44:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: 6T2UO2ZR4KMKWCVFWLXLQBZZHVPLA7NZ5Q5FE2LA4D4QWIMONV5Q) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: EI5HQVD5NURJ7NIFP22ORTDMD4I2NUFYJRWUEIPR4LPAY2SSRAGQ) context deadline exceeded ======LOGS====== ing segments... Dec 28 10:43:47.993729977 (547h14m47.864813022s) [upload=479861]: Got next segment. Inline: false Dec 28 10:43:47.993758972 (547h14m47.864842017s) [upload=479861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:43:47.993763801 (547h14m47.864846856s) [upload=479861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:43:48.239510154 (547h14m48.110593199s) [seg_pos={0 0}, upload=479861]: Begin upload segment... Dec 28 10:43:48.239638047 (547h14m48.110721082s) [seg_pos={0 0}, upload=479861]: Done begin upload segment. Dec 28 10:43:48.239680799 (547h14m48.110763834s) [noise=true , address=145.79.227.13:4802 , piece_id=UPP4RBDB57UTJNTRCZE35WZRCNHED7KWOQGHSKAIS4BGIO45W2WA, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.239701478 (547h14m48.110784523s) [noise=true , address=145.79.227.28:4725 , piece_id=LIUFRUL3HHHJD7LGBALSB4T4EONJOXH22ZIE2EALST7JTW6U2P2Q, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.240279609 (547h14m48.111362644s) [noise=true , address=145.79.227.51:4411 , piece_id=LEGCPWKSOQCINJPJTU5MPL5NKXUNUATLHN2LKOHEEI6UTOZOFRZQ, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.240519035 (547h14m48.11160207s) [noise=true , address=145.79.227.38:4978 , piece_id=PXQN7MTB72NZLJFNR5S46U2CSH7AGD6YIUYTQ7E7HBXWNUHMJEXQ, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.240913406 (547h14m48.111996441s) [noise=true , address=77.90.31.11:4553 , piece_id=NRWBWZCPDVJRSIRSAF46BWQ3JLYHDGYE5LZLDCNYU4XYLYYOBMJQ, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.241221763 (547h14m48.112304798s) [noise=true , address=145.79.227.43:4159 , piece_id=GQUP7QOG2HDMVII54GZI6XJUHCSH3VT3FQEKQYTDNO2HTAXRZHVQ, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.241556029 (547h14m48.112639064s) [noise=true , address=45.91.116.9:4261 , piece_id=W5YWJ26CS62BZR6XB7MLNA7MNCBWMXLMIKTTPQJHT2VB5BRXQOHQ, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.241573172 (547h14m48.112656217s) [noise=true , address=45.91.116.11:4120 , piece_id=KWNGXR6YOP7JSYTS2GPKJGAZTOEVRERX4BEPIYH5QUZSFLIHRGBA, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.241846373 (547h14m48.112929408s) [noise=true , address=45.91.116.6:4974 , piece_id=NNLBTFSPU3EKPZT7MQYHY5NDGHVNJWNNUP7NVEZSNDKLVDYN6XTA, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.242169838 (547h14m48.113252883s) [noise=true , address=45.91.116.4:4940 , piece_id=2UNFUCWWVNFGMJOXNPYEPSCOUCMHRE5BGVLWJVUEGZF6O6WKIJLQ, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.242828072 (547h14m48.113911117s) [noise=true , address=145.79.227.40:4051 , piece_id=HO7ZKUXK5IANQJ2RKG73CSCUVQBU4S3WFRGPOG7NTZUMF3RX7CHA, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.243389651 (547h14m48.114472706s) [noise=true , address=145.79.227.18:4007 , piece_id=55SIKCGSFOBUOFYQUW4QKIRKYFUGNFOQROEX4EWUHSRUL2CY4DIA, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.243854347 (547h14m48.114937402s) [noise=true , address=145.79.227.15:4876 , piece_id=IYSDJFCGLMLZHTRMMQGQHS3ZGESSKGP3KG4ZO4UIP65C6KOTITGQ, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.243959747 (547h14m48.115042792s) [noise=true , address=77.90.31.14:4662 , piece_id=EI5HQVD5NURJ7NIFP22ORTDMD4I2NUFYJRWUEIPR4LPAY2SSRAGQ, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.244477753 (547h14m48.115560798s) [noise=true , address=77.90.31.21:4878 , piece_id=6T2UO2ZR4KMKWCVFWLXLQBZZHVPLA7NZ5Q5FE2LA4D4QWIMONV5Q, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:48.244515265 (547h14m48.1155983s) [noise=true , address=145.79.227.14:4837 , piece_id=LM6GUXOST2SE36356FAVFCPRJEYOKNGLVNNT4Q7QQBERMJR4AQFA, seg_pos={0 0}, upload=479861]: Uploading piece... Dec 28 10:43:49.866345704 (547h14m49.737428739s) [upload=479861]: Next returned nil segment Dec 28 10:43:49.866398145 (547h14m49.73748118s) [upload=479861]: Waiting for error group managing segments... Dec 28 10:43:50.248239055 (547h14m50.1193221s) [noise=true , address=145.79.227.40:4051 , piece_id=HO7ZKUXK5IANQJ2RKG73CSCUVQBU4S3WFRGPOG7NTZUMF3RX7CHA, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.249951525 (547h14m50.12103457s) [noise=true , address=145.79.227.28:4725 , piece_id=LIUFRUL3HHHJD7LGBALSB4T4EONJOXH22ZIE2EALST7JTW6U2P2Q, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.253414459 (547h14m50.124497504s) [noise=true , address=145.79.227.14:4837 , piece_id=LM6GUXOST2SE36356FAVFCPRJEYOKNGLVNNT4Q7QQBERMJR4AQFA, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.255672138 (547h14m50.126755183s) [noise=true , address=145.79.227.43:4159 , piece_id=GQUP7QOG2HDMVII54GZI6XJUHCSH3VT3FQEKQYTDNO2HTAXRZHVQ, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.257090669 (547h14m50.128173704s) [noise=true , address=145.79.227.18:4007 , piece_id=55SIKCGSFOBUOFYQUW4QKIRKYFUGNFOQROEX4EWUHSRUL2CY4DIA, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.263424249 (547h14m50.134507284s) [noise=true , address=145.79.227.51:4411 , piece_id=LEGCPWKSOQCINJPJTU5MPL5NKXUNUATLHN2LKOHEEI6UTOZOFRZQ, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.266553507 (547h14m50.137636542s) [noise=true , address=145.79.227.15:4876 , piece_id=IYSDJFCGLMLZHTRMMQGQHS3ZGESSKGP3KG4ZO4UIP65C6KOTITGQ, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.270217174 (547h14m50.141300219s) [noise=true , address=145.79.227.13:4802 , piece_id=UPP4RBDB57UTJNTRCZE35WZRCNHED7KWOQGHSKAIS4BGIO45W2WA, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.270691116 (547h14m50.141774161s) [noise=true , address=145.79.227.38:4978 , piece_id=PXQN7MTB72NZLJFNR5S46U2CSH7AGD6YIUYTQ7E7HBXWNUHMJEXQ, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.462796297 (547h14m50.333879342s) [noise=true , address=77.90.31.14:4662 , piece_id=EI5HQVD5NURJ7NIFP22ORTDMD4I2NUFYJRWUEIPR4LPAY2SSRAGQ, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.550645136 (547h14m50.421728171s) [noise=true , address=77.90.31.21:4878 , piece_id=6T2UO2ZR4KMKWCVFWLXLQBZZHVPLA7NZ5Q5FE2LA4D4QWIMONV5Q, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.642176397 (547h14m50.513259432s) [noise=true , address=45.91.116.11:4120 , piece_id=KWNGXR6YOP7JSYTS2GPKJGAZTOEVRERX4BEPIYH5QUZSFLIHRGBA, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.660209867 (547h14m50.531292902s) [noise=true , address=45.91.116.9:4261 , piece_id=W5YWJ26CS62BZR6XB7MLNA7MNCBWMXLMIKTTPQJHT2VB5BRXQOHQ, seg_pos={0 0}, upload=479861]: Done uploading piece. err: <nil> Dec 28 10:43:50.660237849 (547h14m50.531320894s) [seg_pos={0 0}, upload=479861]: Segment reached optimal threshold of 13 pieces. Dec 28 10:43:50.660588547 (547h14m50.531671602s) [noise=true , address=45.91.116.4:4940 , piece_id=2UNFUCWWVNFGMJOXNPYEPSCOUCMHRE5BGVLWJVUEGZF6O6WKIJLQ, seg_pos={0 0}, upload=479861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 28 10:43:50.660599238 (547h14m50.531682283s) [noise=true , address=45.91.116.4:4940 , piece_id=2UNFUCWWVNFGMJOXNPYEPSCOUCMHRE5BGVLWJVUEGZF6O6WKIJLQ, seg_pos={0 0}, upload=479861]: Long tail cancellation due to optimal threshold Dec 28 10:43:50.660615238 (547h14m50.531698273s) [noise=true , address=77.90.31.11:4553 , piece_id=NRWBWZCPDVJRSIRSAF46BWQ3JLYHDGYE5LZLDCNYU4XYLYYOBMJQ, seg_pos={0 0}, upload=479861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:54056->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:54056->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54056->77.90.31.11:4553: use of closed network connection Dec 28 10:43:50.660644093 (547h14m50.531727138s) [noise=true , address=77.90.31.11:4553 , piece_id=NRWBWZCPDVJRSIRSAF46BWQ3JLYHDGYE5LZLDCNYU4XYLYYOBMJQ, seg_pos={0 0}, upload=479861]: Long tail cancellation due to optimal threshold Dec 28 10:43:50.660865855 (547h14m50.53194889s) [noise=true , address=45.91.116.6:4974 , piece_id=NNLBTFSPU3EKPZT7MQYHY5NDGHVNJWNNUP7NVEZSNDKLVDYN6XTA, seg_pos={0 0}, upload=479861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:46496->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:46496->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46496->45.91.116.6:4974: use of closed network connection Dec 28 10:43:50.660889400 (547h14m50.531972435s) [noise=true , address=45.91.116.6:4974 , piece_id=NNLBTFSPU3EKPZT7MQYHY5NDGHVNJWNNUP7NVEZSNDKLVDYN6XTA, seg_pos={0 0}, upload=479861]: Long tail cancellation due to optimal threshold Dec 28 10:43:50.660908677 (547h14m50.531991722s) [seg_pos={0 0}, upload=479861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:43:50.661021372 (547h14m50.532104407s) [upload=479861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:43:50.661026040 (547h14m50.532109085s) [upload=479861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:43:50.960995142 (547h14m50.832078217s) [upload=479861]: Done uploading segments... Dec 28 10:43:50.961021773 (547h14m50.832104828s) [upload=479861]: Upload finished. err: <nil> Duration: 30.00318613s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...X.....X..................X..............X.X.........X...X] duration: 5.001131973s trace: https://jaeger.dev.storj.io/trace/3c0c3e1463cd122 FAIL (2025-12-28 10:44:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001157201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...................X..XX.....X..X............XX...........XX] duration: 30.003615873s trace: https://jaeger.dev.storj.io/trace/25fe55b88e992c4 FAIL (2025-12-28 10:43:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: UPSGTJ2PILSIDL5IRODKJ7PPKDHVAUJCJ4H2L4SG7OSOYU3DAXNQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: U5LCQWVPI5D22AS5VHFGW2OJAROQXBZBY4JCHUS7BCBHK2Q2AT3A) context deadline exceeded ======LOGS====== 28 10:43:17.750021586 (547h14m17.621104631s) [upload=479854]: Starting upload Dec 28 10:43:17.750072694 (547h14m17.621155729s) [upload=479854]: Done starting upload Dec 28 10:43:17.750151213 (547h14m17.621234248s) [upload=479854]: Uploading segments... Dec 28 10:43:17.750389557 (547h14m17.621472592s) [upload=479854]: Got next segment. Inline: false Dec 28 10:43:17.750426818 (547h14m17.621509853s) [upload=479854]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:43:17.750431296 (547h14m17.621514331s) [upload=479854]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:43:17.998150246 (547h14m17.869233301s) [seg_pos={0 0}, upload=479854]: Begin upload segment... Dec 28 10:43:17.998203206 (547h14m17.869286251s) [seg_pos={0 0}, upload=479854]: Done begin upload segment. Dec 28 10:43:17.998232031 (547h14m17.869315076s) [noise=true , address=145.79.227.52:4446 , piece_id=YY7UMVPXOSJYF545SRZGGPLJDHE5JRKM7GY4KR55B3I6RC7NYUUQ, seg_pos={0 0}, upload=479854]: Uploading piece... Dec 28 10:43:18.002274679 (547h14m17.873357734s) [noise=true , address=145.79.227.28:4725 , piece_id=UILDSB2O4L3YQKHDEZPKSIJVOYQSE3F3E3NKTVI3UDPNCA2RC67A, seg_pos={0 0}, upload=479854]: Uploading piece... Dec 28 10:43:18.002844464 (547h14m17.873927509s) [noise=true , address=145.79.227.6:4259 , piece_id=ZRP3DZ3HSW3P4EIT5NLIIAY2AVKPYOIXA4R2ZTSUBZAKDWEHCWPA, seg_pos={0 0}, upload=479854]: Uploading piece... Dec 28 10:43:18.003268181 (547h14m17.874351226s) [noise=true , address=145.79.227.47:4266 , piece_id=22ESKR57C7PTNHLE7BX57GBXZ56JNLBAVHV5Y6HH4JC7K3DRE2OQ, seg_pos={0 0}, upload=479854]: Uploading piece... Dec 28 10:43:18.003669164 (547h14m17.874752199s) [noise=true , address=145.79.227.42:4123 , piece_id=46LYFTIQPRBMVJNAIGLQ52EX4CBRZRAJTDX3OYHBXMVSGVQADZFQ, seg_pos={0 0}, upload=479854]: Uploading piece... Dec 28 10:43:18.004097210 (547h14m17.875180245s) [noise=true , address=145.79.227.29:4942 , piece_id=LVMWS5JJKOQ2MITCOV6UUT57M4MEJJRTR6N4XNQUGI74LPKGLRLA, seg_pos={0 0}, upload=479854]: Uploading piece... Dec 28 10:43:18.004507711 (547h14m17.875590746s) [noise=true , address=145.79.227.16:4912 , piece_id=IHRMOFFNBKXQSC7MQ7XJ4LPVDHYS7D6DVNTJCD7K4UUU3HXN5E4A, seg_pos={0 0}, upload=479854]: Uploading piece... Dec 28 10:43:18.005029414 (547h14m17.876112449s) [noise=true , address=145.79.227.38:4978 , piece_id=BEV3JFWPLSRGFVIM7JNRHKRUBLMZIOSA6YRGISRZ5VYB3DPGRUTA, seg_pos={0 0}, upload=479854]: Uploading piece... Dec 28 10:43:18.005938054 (547h14m17.877021099s) [noise=true , address=77.90.31.17:4773 , piece_id=C3FQAIIZSE67JNRWPLDCSAAPTFYF5AODLU3SV3DYFBWW4DFW7DLA, seg_pos={0 0}, upload=479854]: Uploading piece... Dec 28 10:43:18.006459938 (547h14m17.877542973s) [noise=true , address=145.79.227.11:4731 , piece_id=HA2AONOGYLK4QHTJRWLI3DRIU26YJPDR5L5X2VJYWAWXIY6FZL6Q, seg_pos={0 0}, upload=479854]: Uploading piece... Dec 28 10:43:18.007175842 (547h14m17.878258897s) [noise=true , address=77.90.31.13:4627 , piece_id=UPSGTJ2PILSIDL5IRODKJ7PPKDHVAUJCJ4H2L4SG7OSOYU3DAXNQ, seg_pos={0 0}, upload=479854]: Uploading piece... Dec 28 10:43:18.007908677 (547h14m17.878991722s) [noise=true , address=45.91.116.9:4264 , piece_id=U5LCQWVPI5D22AS5VHFGW2OJAROQXBZBY4JCHUS7BCBHK2Q2AT3A, seg_pos={0 0}, upload=479854]: Uploading piece... Dec 28 10:43:18.008683833 (547h14m17.879766888s) [noise=true , address=45.91.116.14:4191 , piece_id=QZQZLCSZJ5XF53P2BBNS4R2WYHY2276VPAZCDUXSW7TRJMOGIWXA, seg_pos={0 0}, upload=479854]: Uploading piece... Dec 28 10:43:18.009347006 (547h14m17.880430051s) [noise=true , address=145.79.227.3:4150 , piece_id=ANUSFZ2RGZ2HKGPCPJIFNSS2JKIJCP2VEP3B534EQ72RMVV5BXQA, seg_pos={0 0}, upload=479854]: Uploading piece... Dec 28 10:43:18.010453182 (547h14m17.881536217s) [noise=true , address=45.91.116.10:4083 , piece_id=TQXAK7HQ7UKSMNLV7QF7Q6BXFI5RH3FZFSSVAZYKA36AYWKJBTWA, seg_pos={0 0}, upload=479854]: Uploading piece... Dec 28 10:43:18.011138547 (547h14m17.882221593s) [noise=true , address=145.79.227.5:4223 , piece_id=WMDMUJXAJUHHM7IZAV4NABTR5CJ4DHF7BZJIWGEMWITD6IUTARPA, seg_pos={0 0}, upload=479854]: Uploading piece... Dec 28 10:43:19.622425361 (547h14m19.493508396s) [upload=479854]: Next returned nil segment Dec 28 10:43:19.622438155 (547h14m19.4935212s) [upload=479854]: Waiting for error group managing segments... Dec 28 10:43:20.007331146 (547h14m19.878414181s) [noise=true , address=145.79.227.3:4150 , piece_id=ANUSFZ2RGZ2HKGPCPJIFNSS2JKIJCP2VEP3B534EQ72RMVV5BXQA, seg_pos={0 0}, upload=479854]: Done uploading piece. err: <nil> Dec 28 10:43:20.011072520 (547h14m19.882155555s) [noise=true , address=145.79.227.38:4978 , piece_id=BEV3JFWPLSRGFVIM7JNRHKRUBLMZIOSA6YRGISRZ5VYB3DPGRUTA, seg_pos={0 0}, upload=479854]: Done uploading piece. err: <nil> Dec 28 10:43:20.011359527 (547h14m19.882442572s) [noise=true , address=145.79.227.28:4725 , piece_id=UILDSB2O4L3YQKHDEZPKSIJVOYQSE3F3E3NKTVI3UDPNCA2RC67A, seg_pos={0 0}, upload=479854]: Done uploading piece. err: <nil> Dec 28 10:43:20.013756792 (547h14m19.884839827s) [noise=true , address=145.79.227.52:4446 , piece_id=YY7UMVPXOSJYF545SRZGGPLJDHE5JRKM7GY4KR55B3I6RC7NYUUQ, seg_pos={0 0}, upload=479854]: Done uploading piece. err: <nil> Dec 28 10:43:20.014864391 (547h14m19.885947436s) [noise=true , address=145.79.227.11:4731 , piece_id=HA2AONOGYLK4QHTJRWLI3DRIU26YJPDR5L5X2VJYWAWXIY6FZL6Q, seg_pos={0 0}, upload=479854]: Done uploading piece. err: <nil> Dec 28 10:43:20.015053190 (547h14m19.886136235s) [noise=true , address=145.79.227.42:4123 , piece_id=46LYFTIQPRBMVJNAIGLQ52EX4CBRZRAJTDX3OYHBXMVSGVQADZFQ, seg_pos={0 0}, upload=479854]: Done uploading piece. err: <nil> Dec 28 10:43:20.017612313 (547h14m19.888695348s) [noise=true , address=145.79.227.6:4259 , piece_id=ZRP3DZ3HSW3P4EIT5NLIIAY2AVKPYOIXA4R2ZTSUBZAKDWEHCWPA, seg_pos={0 0}, upload=479854]: Done uploading piece. err: <nil> Dec 28 10:43:20.022341859 (547h14m19.893424894s) [noise=true , address=145.79.227.5:4223 , piece_id=WMDMUJXAJUHHM7IZAV4NABTR5CJ4DHF7BZJIWGEMWITD6IUTARPA, seg_pos={0 0}, upload=479854]: Done uploading piece. err: <nil> Dec 28 10:43:20.025475086 (547h14m19.896558131s) [noise=true , address=145.79.227.29:4942 , piece_id=LVMWS5JJKOQ2MITCOV6UUT57M4MEJJRTR6N4XNQUGI74LPKGLRLA, seg_pos={0 0}, upload=479854]: Done uploading piece. err: <nil> Dec 28 10:43:20.036221909 (547h14m19.907304954s) [noise=true , address=145.79.227.16:4912 , piece_id=IHRMOFFNBKXQSC7MQ7XJ4LPVDHYS7D6DVNTJCD7K4UUU3HXN5E4A, seg_pos={0 0}, upload=479854]: Done uploading piece. err: <nil> Dec 28 10:43:20.047112516 (547h14m19.918195552s) [noise=true , address=145.79.227.47:4266 , piece_id=22ESKR57C7PTNHLE7BX57GBXZ56JNLBAVHV5Y6HH4JC7K3DRE2OQ, seg_pos={0 0}, upload=479854]: Done uploading piece. err: <nil> Dec 28 10:43:20.430555567 (547h14m20.301638612s) [noise=true , address=45.91.116.9:4264 , piece_id=U5LCQWVPI5D22AS5VHFGW2OJAROQXBZBY4JCHUS7BCBHK2Q2AT3A, seg_pos={0 0}, upload=479854]: Done uploading piece. err: <nil> Dec 28 10:43:20.483891197 (547h14m20.354974232s) [noise=true , address=77.90.31.13:4627 , piece_id=UPSGTJ2PILSIDL5IRODKJ7PPKDHVAUJCJ4H2L4SG7OSOYU3DAXNQ, seg_pos={0 0}, upload=479854]: Done uploading piece. err: <nil> Dec 28 10:43:20.483912338 (547h14m20.354995373s) [seg_pos={0 0}, upload=479854]: Segment reached optimal threshold of 13 pieces. Dec 28 10:43:20.484080267 (547h14m20.355163302s) [noise=true , address=77.90.31.17:4773 , piece_id=C3FQAIIZSE67JNRWPLDCSAAPTFYF5AODLU3SV3DYFBWW4DFW7DLA, seg_pos={0 0}, upload=479854]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:59588->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:59588->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59588->77.90.31.17:4773: use of closed network connection Dec 28 10:43:20.484085948 (547h14m20.355168984s) [noise=true , address=77.90.31.17:4773 , piece_id=C3FQAIIZSE67JNRWPLDCSAAPTFYF5AODLU3SV3DYFBWW4DFW7DLA, seg_pos={0 0}, upload=479854]: Long tail cancellation due to optimal threshold Dec 28 10:43:20.484234932 (547h14m20.355317977s) [noise=true , address=45.91.116.14:4191 , piece_id=QZQZLCSZJ5XF53P2BBNS4R2WYHY2276VPAZCDUXSW7TRJMOGIWXA, seg_pos={0 0}, upload=479854]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 28 10:43:20.484245842 (547h14m20.355328887s) [noise=true , address=45.91.116.14:4191 , piece_id=QZQZLCSZJ5XF53P2BBNS4R2WYHY2276VPAZCDUXSW7TRJMOGIWXA, seg_pos={0 0}, upload=479854]: Long tail cancellation due to optimal threshold Dec 28 10:43:20.484252425 (547h14m20.35533546s) [noise=true , address=45.91.116.10:4083 , piece_id=TQXAK7HQ7UKSMNLV7QF7Q6BXFI5RH3FZFSSVAZYKA36AYWKJBTWA, seg_pos={0 0}, upload=479854]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 28 10:43:20.484258798 (547h14m20.355341833s) [noise=true , address=45.91.116.10:4083 , piece_id=TQXAK7HQ7UKSMNLV7QF7Q6BXFI5RH3FZFSSVAZYKA36AYWKJBTWA, seg_pos={0 0}, upload=479854]: Long tail cancellation due to optimal threshold Dec 28 10:43:20.484339351 (547h14m20.355422396s) [seg_pos={0 0}, upload=479854]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:43:20.484408502 (547h14m20.355491537s) [upload=479854]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:43:20.484413852 (547h14m20.355496888s) [upload=479854]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:43:20.769056330 (547h14m20.640139375s) [upload=479854]: Done uploading segments... Dec 28 10:43:20.769126764 (547h14m20.640209799s) [upload=479854]: Upload finished. err: <nil> Duration: 30.003639809s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....X.....X..................X..............X.X.........X...] duration: 3.409240162s trace: https://jaeger.dev.storj.io/trace/42d15caac6419534 FAIL (2025-12-28 10:43:01 UTC) orbiter warning: context deadline exceeded Duration: 3.409276281s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X...................X..XX.....X..X............XX...........X] duration: 30.003039515s trace: https://jaeger.dev.storj.io/trace/1a68b99c7dd6fb4d FAIL (2025-12-28 10:42:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 10:42:04.410581530 (547h13m4.281664586s) [upload=479836]: Starting upload Dec 28 10:42:04.410745131 (547h13m4.281828176s) [upload=479836]: Done starting upload Dec 28 10:42:04.410835824 (547h13m4.281918869s) [upload=479836]: Uploading segments... Dec 28 10:42:04.411077635 (547h13m4.28216067s) [upload=479836]: Got next segment. Inline: false Dec 28 10:42:04.411131266 (547h13m4.282214312s) [upload=479836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:42:04.411138671 (547h13m4.282221716s) [upload=479836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:42:04.657053765 (547h13m4.52813682s) [seg_pos={0 0}, upload=479836]: Begin upload segment... Dec 28 10:42:04.657121664 (547h13m4.528204709s) [seg_pos={0 0}, upload=479836]: Done begin upload segment. Dec 28 10:42:04.657179605 (547h13m4.52826265s) [noise=true , address=145.79.227.47:4267 , piece_id=YBU2WJJXQ4FCIBSEF3YRHP7RPPCZV4IMJ3AYLAVHMOAFU4TPLA5A, seg_pos={0 0}, upload=479836]: Uploading piece... Dec 28 10:42:04.657269666 (547h13m4.528352711s) [noise=true , address=145.79.227.42:4122 , piece_id=FXN5BB4BGTJQNHMIX5CIA3UXODERBHQX4Z4HOZJTU2NI2MBOTU6Q, seg_pos={0 0}, upload=479836]: Uploading piece... Dec 28 10:42:04.658009645 (547h13m4.52909269s) [noise=true , address=145.79.227.10:4401 , piece_id=36L7EHVFIHXCSWFD4SW7SSBLEQ33TJIUIA37KO2BYMGKSUKKJLFA, seg_pos={0 0}, upload=479836]: Uploading piece... Dec 28 10:42:04.658082985 (547h13m4.52916603s) [noise=true , address=145.79.227.7:4294 , piece_id=4ZAW6TCRMTF7AD7FYE7UZVAWHO5AAU772VMNIS3HYNFPZQLP53HQ, seg_pos={0 0}, upload=479836]: Uploading piece... Dec 28 10:42:04.658538212 (547h13m4.529621257s) [noise=true , address=77.90.31.21:4877 , piece_id=CGZCYEZXEKQJZNLBIZYLKSSTNXBJ7RRST732PW7KJTHXR6WGFA6A, seg_pos={0 0}, upload=479836]: Uploading piece... Dec 28 10:42:04.658627732 (547h13m4.529710767s) [noise=true , address=145.79.227.41:4085 , piece_id=QO5MFXKN4DSFXXHPWRORTS3GVAXZ65Y2PLTQ2MYNQJ77A2P7GDNA, seg_pos={0 0}, upload=479836]: Uploading piece... Dec 28 10:42:04.659432324 (547h13m4.530515369s) [noise=true , address=45.91.116.8:4046 , piece_id=FPZVRBHAYA6D7GSXYNTHUCCX54XHBM2XURPB3B3BQSJFC2VAN2NQ, seg_pos={0 0}, upload=479836]: Uploading piece... Dec 28 10:42:04.659712447 (547h13m4.530795492s) [noise=true , address=145.79.227.2:4115 , piece_id=TWNU6RVIR5RDGOZIVY6WXOM5YG3NMN5WOWLWVE7TJMLXR3P35T3Q, seg_pos={0 0}, upload=479836]: Uploading piece... Dec 28 10:42:04.660221467 (547h13m4.531304512s) [noise=true , address=145.79.227.51:4410 , piece_id=B47ALOQ2YPZYDNPC27XE6R6DLXSKGRTSXBW7ARS6CTLEKCOWDOPA, seg_pos={0 0}, upload=479836]: Uploading piece... Dec 28 10:42:04.660288935 (547h13m4.53137198s) [noise=true , address=145.79.227.3:4151 , piece_id=3VHVTWN6RO5BOJG4IOJKMT62AJK74TSRYM6QF257J3F74THB3QKQ, seg_pos={0 0}, upload=479836]: Uploading piece... Dec 28 10:42:04.660723552 (547h13m4.531806597s) [noise=true , address=145.79.227.11:4732 , piece_id=O4VSRKDYSW3FEZVTQGZOLDY6666GNZQFW6RPPYOHSIPIH3WMNGRA, seg_pos={0 0}, upload=479836]: Uploading piece... Dec 28 10:42:04.661106962 (547h13m4.532190007s) [noise=true , address=77.90.31.14:4664 , piece_id=AXAR5CFAIKQYXQICZR6A3FK6AY5VPW3XMVMNABLAXTLNOIZGDXYQ, seg_pos={0 0}, upload=479836]: Uploading piece... Dec 28 10:42:04.661184841 (547h13m4.532267887s) [noise=true , address=145.79.227.14:4837 , piece_id=BHWPIQAC3YPQ5LY6V3IG7LELS6GXA6VMH6JAO6ALS2UBCOU5Q55Q, seg_pos={0 0}, upload=479836]: Uploading piece... Dec 28 10:42:04.661707697 (547h13m4.532790742s) [noise=true , address=45.91.116.9:4262 , piece_id=DYNX2BMMECPNFTM7R4L5INAQMOE4RM2JNACTCIWQP2S7SJXNST6A, seg_pos={0 0}, upload=479836]: Uploading piece... Dec 28 10:42:04.662112577 (547h13m4.533195622s) [noise=true , address=145.79.227.16:4911 , piece_id=2TM7MOROZL2CIKAF275M6322O2IXDZX7Q2WLZLRKQLCZCVYYF34A, seg_pos={0 0}, upload=479836]: Uploading piece... Dec 28 10:42:04.663211180 (547h13m4.534294215s) [noise=true , address=45.91.116.11:4117 , piece_id=UI3UHWCRFPMJ2ULV45D7PGIWRE5J4QBXYT3KHA2NXEF55ADJDXBA, seg_pos={0 0}, upload=479836]: Uploading piece... Dec 28 10:42:06.371709400 (547h13m6.242792445s) [upload=479836]: Next returned nil segment Dec 28 10:42:06.371725340 (547h13m6.242808385s) [upload=479836]: Waiting for error group managing segments... Dec 28 10:42:06.653902113 (547h13m6.524985148s) [noise=true , address=145.79.227.2:4115 , piece_id=TWNU6RVIR5RDGOZIVY6WXOM5YG3NMN5WOWLWVE7TJMLXR3P35T3Q, seg_pos={0 0}, upload=479836]: Done uploading piece. err: <nil> Dec 28 10:42:06.663330956 (547h13m6.534414001s) [noise=true , address=145.79.227.41:4085 , piece_id=QO5MFXKN4DSFXXHPWRORTS3GVAXZ65Y2PLTQ2MYNQJ77A2P7GDNA, seg_pos={0 0}, upload=479836]: Done uploading piece. err: <nil> Dec 28 10:42:06.664515792 (547h13m6.535598837s) [noise=true , address=145.79.227.11:4732 , piece_id=O4VSRKDYSW3FEZVTQGZOLDY6666GNZQFW6RPPYOHSIPIH3WMNGRA, seg_pos={0 0}, upload=479836]: Done uploading piece. err: <nil> Dec 28 10:42:06.665273625 (547h13m6.536356671s) [noise=true , address=145.79.227.16:4911 , piece_id=2TM7MOROZL2CIKAF275M6322O2IXDZX7Q2WLZLRKQLCZCVYYF34A, seg_pos={0 0}, upload=479836]: Done uploading piece. err: <nil> Dec 28 10:42:06.672149649 (547h13m6.543232694s) [noise=true , address=145.79.227.42:4122 , piece_id=FXN5BB4BGTJQNHMIX5CIA3UXODERBHQX4Z4HOZJTU2NI2MBOTU6Q, seg_pos={0 0}, upload=479836]: Done uploading piece. err: <nil> Dec 28 10:42:06.681370907 (547h13m6.552453952s) [noise=true , address=145.79.227.47:4267 , piece_id=YBU2WJJXQ4FCIBSEF3YRHP7RPPCZV4IMJ3AYLAVHMOAFU4TPLA5A, seg_pos={0 0}, upload=479836]: Done uploading piece. err: <nil> Dec 28 10:42:06.682941407 (547h13m6.554024442s) [noise=true , address=145.79.227.51:4410 , piece_id=B47ALOQ2YPZYDNPC27XE6R6DLXSKGRTSXBW7ARS6CTLEKCOWDOPA, seg_pos={0 0}, upload=479836]: Done uploading piece. err: <nil> Dec 28 10:42:06.687343731 (547h13m6.558426776s) [noise=true , address=145.79.227.14:4837 , piece_id=BHWPIQAC3YPQ5LY6V3IG7LELS6GXA6VMH6JAO6ALS2UBCOU5Q55Q, seg_pos={0 0}, upload=479836]: Done uploading piece. err: <nil> Dec 28 10:42:06.689325124 (547h13m6.560408159s) [noise=true , address=145.79.227.7:4294 , piece_id=4ZAW6TCRMTF7AD7FYE7UZVAWHO5AAU772VMNIS3HYNFPZQLP53HQ, seg_pos={0 0}, upload=479836]: Done uploading piece. err: <nil> Dec 28 10:42:06.691912220 (547h13m6.562995255s) [noise=true , address=145.79.227.3:4151 , piece_id=3VHVTWN6RO5BOJG4IOJKMT62AJK74TSRYM6QF257J3F74THB3QKQ, seg_pos={0 0}, upload=479836]: Done uploading piece. err: <nil> Dec 28 10:42:06.694942780 (547h13m6.566025815s) [noise=true , address=145.79.227.10:4401 , piece_id=36L7EHVFIHXCSWFD4SW7SSBLEQ33TJIUIA37KO2BYMGKSUKKJLFA, seg_pos={0 0}, upload=479836]: Done uploading piece. err: <nil> Dec 28 10:42:06.859066831 (547h13m6.730149866s) [noise=true , address=77.90.31.14:4664 , piece_id=AXAR5CFAIKQYXQICZR6A3FK6AY5VPW3XMVMNABLAXTLNOIZGDXYQ, seg_pos={0 0}, upload=479836]: Done uploading piece. err: <nil> Dec 28 10:42:06.910144340 (547h13m6.781227375s) [noise=true , address=77.90.31.21:4877 , piece_id=CGZCYEZXEKQJZNLBIZYLKSSTNXBJ7RRST732PW7KJTHXR6WGFA6A, seg_pos={0 0}, upload=479836]: Done uploading piece. err: <nil> Dec 28 10:42:06.910164459 (547h13m6.781247494s) [seg_pos={0 0}, upload=479836]: Segment reached optimal threshold of 13 pieces. Dec 28 10:42:06.910540335 (547h13m6.78162338s) [noise=true , address=45.91.116.9:4262 , piece_id=DYNX2BMMECPNFTM7R4L5INAQMOE4RM2JNACTCIWQP2S7SJXNST6A, seg_pos={0 0}, upload=479836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 28 10:42:06.910548991 (547h13m6.781632026s) [noise=true , address=45.91.116.9:4262 , piece_id=DYNX2BMMECPNFTM7R4L5INAQMOE4RM2JNACTCIWQP2S7SJXNST6A, seg_pos={0 0}, upload=479836]: Long tail cancellation due to optimal threshold Dec 28 10:42:06.910641979 (547h13m6.781725024s) [noise=true , address=45.91.116.11:4117 , piece_id=UI3UHWCRFPMJ2ULV45D7PGIWRE5J4QBXYT3KHA2NXEF55ADJDXBA, seg_pos={0 0}, upload=479836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 28 10:42:06.910658490 (547h13m6.781741525s) [noise=true , address=45.91.116.11:4117 , piece_id=UI3UHWCRFPMJ2ULV45D7PGIWRE5J4QBXYT3KHA2NXEF55ADJDXBA, seg_pos={0 0}, upload=479836]: Long tail cancellation due to optimal threshold Dec 28 10:42:06.910841728 (547h13m6.781924773s) [noise=true , address=45.91.116.8:4046 , piece_id=FPZVRBHAYA6D7GSXYNTHUCCX54XHBM2XURPB3B3BQSJFC2VAN2NQ, seg_pos={0 0}, upload=479836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 28 10:42:06.910851618 (547h13m6.781934663s) [noise=true , address=45.91.116.8:4046 , piece_id=FPZVRBHAYA6D7GSXYNTHUCCX54XHBM2XURPB3B3BQSJFC2VAN2NQ, seg_pos={0 0}, upload=479836]: Long tail cancellation due to optimal threshold Dec 28 10:42:06.910877617 (547h13m6.781960663s) [seg_pos={0 0}, upload=479836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:42:06.910973058 (547h13m6.782056104s) [upload=479836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:42:06.910979641 (547h13m6.782062696s) [upload=479836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:42:07.215703046 (547h13m7.086786091s) [upload=479836]: Done uploading segments... Dec 28 10:42:07.215729427 (547h13m7.086812462s) [upload=479836]: Upload finished. err: <nil> Duration: 30.003063312s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.....X.....X..................X..............X.X.........X..] duration: 4.007304405s trace: https://jaeger.dev.storj.io/trace/42d15caac6419534 FAIL (2025-12-28 10:41:55 UTC) orbiter warning: context deadline exceeded Duration: 4.007361154s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X...................X..XX.....X..X............XX...........] duration: 5.648293788s trace: https://jaeger.dev.storj.io/trace/12fdeb4fc6be55f2 FAIL (2025-12-28 10:41:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A) context deadline exceeded ======LOGS====== Dec 28 10:32:09.171162516 (547h3m9.042245551s) [upload=479693]: Starting upload Dec 28 10:32:09.171211930 (547h3m9.042294965s) [upload=479693]: Done starting upload Dec 28 10:32:09.171389138 (547h3m9.042472173s) [upload=479693]: Uploading segments... Dec 28 10:32:09.171420808 (547h3m9.042503843s) [upload=479693]: Got next segment. Inline: false Dec 28 10:32:09.171446587 (547h3m9.042529622s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:32:09.171451076 (547h3m9.042534121s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:32:09.415778928 (547h3m9.286861973s) [seg_pos={0 0}, upload=479693]: Begin upload segment... Dec 28 10:32:09.415823763 (547h3m9.286906798s) [seg_pos={0 0}, upload=479693]: Done begin upload segment. Dec 28 10:32:09.415852628 (547h3m9.286935673s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.415915609 (547h3m9.286998644s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416427983 (547h3m9.287511019s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416839006 (547h3m9.287922041s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416935741 (547h3m9.288018786s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417242896 (547h3m9.288325941s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417527407 (547h3m9.288610442s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417826456 (547h3m9.288909491s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417930104 (547h3m9.289013149s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418096461 (547h3m9.289179506s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418451897 (547h3m9.289534942s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419025139 (547h3m9.290108184s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419682581 (547h3m9.290765626s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.420365671 (547h3m9.291448706s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421011522 (547h3m9.292094557s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421697077 (547h3m9.292780122s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:11.133534367 (547h3m11.004617412s) [upload=479693]: Next returned nil segment Dec 28 10:32:11.133569233 (547h3m11.004652278s) [upload=479693]: Waiting for error group managing segments... Dec 28 10:32:11.421476958 (547h3m11.292559993s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.423163530 (547h3m11.294246575s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.428115299 (547h3m11.299198334s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.436758778 (547h3m11.307841823s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439394376 (547h3m11.310477411s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439861225 (547h3m11.31094427s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.442937283 (547h3m11.314020318s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.444228622 (547h3m11.315311667s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.452679474 (547h3m11.323762519s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.457542493 (547h3m11.328625538s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.708367176 (547h3m11.579450221s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.716026030 (547h3m11.587109075s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434215195 (547h3m12.30529824s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434324403 (547h3m12.305407438s) [seg_pos={0 0}, upload=479693]: Segment reached optimal threshold of 13 pieces. Dec 28 10:32:12.434515456 (547h3m12.305598501s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434521548 (547h3m12.305604593s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434710859 (547h3m12.305793894s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434716900 (547h3m12.305799945s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434797904 (547h3m12.305880949s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434813053 (547h3m12.305896088s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434845295 (547h3m12.30592833s) [seg_pos={0 0}, upload=479693]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:32:12.434957198 (547h3m12.306040233s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:32:12.434962488 (547h3m12.306045523s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:32:12.725143372 (547h3m12.596226407s) [upload=479693]: Done uploading segments... Dec 28 10:32:12.725163410 (547h3m12.596246445s) [upload=479693]: Upload finished. err: <nil> Duration: 5.648319997s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......X.....X..................X..............X.X.........X.] duration: 3.587707754s trace: https://jaeger.dev.storj.io/trace/42d15caac6419534 FAIL (2025-12-28 10:41:08 UTC) orbiter warning: context deadline exceeded Duration: 3.587795632s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X...................X..XX.....X..X............XX..........] duration: 5.718886215s trace: https://jaeger.dev.storj.io/trace/12fdeb4fc6be55f2 FAIL (2025-12-28 10:40:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A) context deadline exceeded ======LOGS====== Dec 28 10:32:09.171162516 (547h3m9.042245551s) [upload=479693]: Starting upload Dec 28 10:32:09.171211930 (547h3m9.042294965s) [upload=479693]: Done starting upload Dec 28 10:32:09.171389138 (547h3m9.042472173s) [upload=479693]: Uploading segments... Dec 28 10:32:09.171420808 (547h3m9.042503843s) [upload=479693]: Got next segment. Inline: false Dec 28 10:32:09.171446587 (547h3m9.042529622s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:32:09.171451076 (547h3m9.042534121s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:32:09.415778928 (547h3m9.286861973s) [seg_pos={0 0}, upload=479693]: Begin upload segment... Dec 28 10:32:09.415823763 (547h3m9.286906798s) [seg_pos={0 0}, upload=479693]: Done begin upload segment. Dec 28 10:32:09.415852628 (547h3m9.286935673s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.415915609 (547h3m9.286998644s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416427983 (547h3m9.287511019s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416839006 (547h3m9.287922041s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416935741 (547h3m9.288018786s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417242896 (547h3m9.288325941s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417527407 (547h3m9.288610442s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417826456 (547h3m9.288909491s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417930104 (547h3m9.289013149s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418096461 (547h3m9.289179506s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418451897 (547h3m9.289534942s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419025139 (547h3m9.290108184s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419682581 (547h3m9.290765626s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.420365671 (547h3m9.291448706s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421011522 (547h3m9.292094557s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421697077 (547h3m9.292780122s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:11.133534367 (547h3m11.004617412s) [upload=479693]: Next returned nil segment Dec 28 10:32:11.133569233 (547h3m11.004652278s) [upload=479693]: Waiting for error group managing segments... Dec 28 10:32:11.421476958 (547h3m11.292559993s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.423163530 (547h3m11.294246575s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.428115299 (547h3m11.299198334s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.436758778 (547h3m11.307841823s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439394376 (547h3m11.310477411s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439861225 (547h3m11.31094427s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.442937283 (547h3m11.314020318s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.444228622 (547h3m11.315311667s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.452679474 (547h3m11.323762519s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.457542493 (547h3m11.328625538s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.708367176 (547h3m11.579450221s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.716026030 (547h3m11.587109075s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434215195 (547h3m12.30529824s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434324403 (547h3m12.305407438s) [seg_pos={0 0}, upload=479693]: Segment reached optimal threshold of 13 pieces. Dec 28 10:32:12.434515456 (547h3m12.305598501s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434521548 (547h3m12.305604593s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434710859 (547h3m12.305793894s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434716900 (547h3m12.305799945s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434797904 (547h3m12.305880949s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434813053 (547h3m12.305896088s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434845295 (547h3m12.30592833s) [seg_pos={0 0}, upload=479693]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:32:12.434957198 (547h3m12.306040233s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:32:12.434962488 (547h3m12.306045523s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:32:12.725143372 (547h3m12.596226407s) [upload=479693]: Done uploading segments... Dec 28 10:32:12.725163410 (547h3m12.596246445s) [upload=479693]: Upload finished. err: <nil> Duration: 5.71891431s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.......X.....X..................X..............X.X.........X] duration: 5.001242753s trace: https://jaeger.dev.storj.io/trace/42d15caac6419534 FAIL (2025-12-28 10:40:14 UTC) orbiter warning: context deadline exceeded Duration: 5.001268192s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X...................X..XX.....X..X............XX.........] duration: 5.637291508s trace: https://jaeger.dev.storj.io/trace/12fdeb4fc6be55f2 FAIL (2025-12-28 10:39:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A) context deadline exceeded ======LOGS====== Dec 28 10:32:09.171162516 (547h3m9.042245551s) [upload=479693]: Starting upload Dec 28 10:32:09.171211930 (547h3m9.042294965s) [upload=479693]: Done starting upload Dec 28 10:32:09.171389138 (547h3m9.042472173s) [upload=479693]: Uploading segments... Dec 28 10:32:09.171420808 (547h3m9.042503843s) [upload=479693]: Got next segment. Inline: false Dec 28 10:32:09.171446587 (547h3m9.042529622s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:32:09.171451076 (547h3m9.042534121s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:32:09.415778928 (547h3m9.286861973s) [seg_pos={0 0}, upload=479693]: Begin upload segment... Dec 28 10:32:09.415823763 (547h3m9.286906798s) [seg_pos={0 0}, upload=479693]: Done begin upload segment. Dec 28 10:32:09.415852628 (547h3m9.286935673s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.415915609 (547h3m9.286998644s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416427983 (547h3m9.287511019s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416839006 (547h3m9.287922041s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416935741 (547h3m9.288018786s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417242896 (547h3m9.288325941s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417527407 (547h3m9.288610442s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417826456 (547h3m9.288909491s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417930104 (547h3m9.289013149s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418096461 (547h3m9.289179506s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418451897 (547h3m9.289534942s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419025139 (547h3m9.290108184s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419682581 (547h3m9.290765626s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.420365671 (547h3m9.291448706s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421011522 (547h3m9.292094557s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421697077 (547h3m9.292780122s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:11.133534367 (547h3m11.004617412s) [upload=479693]: Next returned nil segment Dec 28 10:32:11.133569233 (547h3m11.004652278s) [upload=479693]: Waiting for error group managing segments... Dec 28 10:32:11.421476958 (547h3m11.292559993s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.423163530 (547h3m11.294246575s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.428115299 (547h3m11.299198334s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.436758778 (547h3m11.307841823s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439394376 (547h3m11.310477411s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439861225 (547h3m11.31094427s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.442937283 (547h3m11.314020318s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.444228622 (547h3m11.315311667s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.452679474 (547h3m11.323762519s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.457542493 (547h3m11.328625538s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.708367176 (547h3m11.579450221s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.716026030 (547h3m11.587109075s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434215195 (547h3m12.30529824s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434324403 (547h3m12.305407438s) [seg_pos={0 0}, upload=479693]: Segment reached optimal threshold of 13 pieces. Dec 28 10:32:12.434515456 (547h3m12.305598501s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434521548 (547h3m12.305604593s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434710859 (547h3m12.305793894s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434716900 (547h3m12.305799945s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434797904 (547h3m12.305880949s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434813053 (547h3m12.305896088s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434845295 (547h3m12.30592833s) [seg_pos={0 0}, upload=479693]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:32:12.434957198 (547h3m12.306040233s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:32:12.434962488 (547h3m12.306045523s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:32:12.725143372 (547h3m12.596226407s) [upload=479693]: Done uploading segments... Dec 28 10:32:12.725163410 (547h3m12.596246445s) [upload=479693]: Upload finished. err: <nil> Duration: 5.63732428s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [........X.....X..................X..............X.X.........] duration: 3.66170653s trace: https://jaeger.dev.storj.io/trace/57df8e96b91a6205 FAIL (2025-12-28 10:39:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.661732339s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X...................X..XX.....X..X............XX........] duration: 5.837708598s trace: https://jaeger.dev.storj.io/trace/12fdeb4fc6be55f2 FAIL (2025-12-28 10:39:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A) context deadline exceeded ======LOGS====== Dec 28 10:32:09.171162516 (547h3m9.042245551s) [upload=479693]: Starting upload Dec 28 10:32:09.171211930 (547h3m9.042294965s) [upload=479693]: Done starting upload Dec 28 10:32:09.171389138 (547h3m9.042472173s) [upload=479693]: Uploading segments... Dec 28 10:32:09.171420808 (547h3m9.042503843s) [upload=479693]: Got next segment. Inline: false Dec 28 10:32:09.171446587 (547h3m9.042529622s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:32:09.171451076 (547h3m9.042534121s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:32:09.415778928 (547h3m9.286861973s) [seg_pos={0 0}, upload=479693]: Begin upload segment... Dec 28 10:32:09.415823763 (547h3m9.286906798s) [seg_pos={0 0}, upload=479693]: Done begin upload segment. Dec 28 10:32:09.415852628 (547h3m9.286935673s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.415915609 (547h3m9.286998644s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416427983 (547h3m9.287511019s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416839006 (547h3m9.287922041s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416935741 (547h3m9.288018786s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417242896 (547h3m9.288325941s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417527407 (547h3m9.288610442s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417826456 (547h3m9.288909491s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417930104 (547h3m9.289013149s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418096461 (547h3m9.289179506s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418451897 (547h3m9.289534942s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419025139 (547h3m9.290108184s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419682581 (547h3m9.290765626s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.420365671 (547h3m9.291448706s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421011522 (547h3m9.292094557s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421697077 (547h3m9.292780122s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:11.133534367 (547h3m11.004617412s) [upload=479693]: Next returned nil segment Dec 28 10:32:11.133569233 (547h3m11.004652278s) [upload=479693]: Waiting for error group managing segments... Dec 28 10:32:11.421476958 (547h3m11.292559993s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.423163530 (547h3m11.294246575s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.428115299 (547h3m11.299198334s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.436758778 (547h3m11.307841823s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439394376 (547h3m11.310477411s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439861225 (547h3m11.31094427s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.442937283 (547h3m11.314020318s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.444228622 (547h3m11.315311667s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.452679474 (547h3m11.323762519s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.457542493 (547h3m11.328625538s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.708367176 (547h3m11.579450221s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.716026030 (547h3m11.587109075s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434215195 (547h3m12.30529824s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434324403 (547h3m12.305407438s) [seg_pos={0 0}, upload=479693]: Segment reached optimal threshold of 13 pieces. Dec 28 10:32:12.434515456 (547h3m12.305598501s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434521548 (547h3m12.305604593s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434710859 (547h3m12.305793894s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434716900 (547h3m12.305799945s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434797904 (547h3m12.305880949s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434813053 (547h3m12.305896088s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434845295 (547h3m12.30592833s) [seg_pos={0 0}, upload=479693]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:32:12.434957198 (547h3m12.306040233s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:32:12.434962488 (547h3m12.306045523s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:32:12.725143372 (547h3m12.596226407s) [upload=479693]: Done uploading segments... Dec 28 10:32:12.725163410 (547h3m12.596246445s) [upload=479693]: Upload finished. err: <nil> Duration: 5.83773659s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X........X.....X..................X..............X.X........] duration: 3.413997399s trace: https://jaeger.dev.storj.io/trace/57df8e96b91a6205 FAIL (2025-12-28 10:38:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.414249369s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X...................X..XX.....X..X............XX.......] duration: 5.929829585s trace: https://jaeger.dev.storj.io/trace/12fdeb4fc6be55f2 FAIL (2025-12-28 10:38:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A) context deadline exceeded ======LOGS====== Dec 28 10:32:09.171162516 (547h3m9.042245551s) [upload=479693]: Starting upload Dec 28 10:32:09.171211930 (547h3m9.042294965s) [upload=479693]: Done starting upload Dec 28 10:32:09.171389138 (547h3m9.042472173s) [upload=479693]: Uploading segments... Dec 28 10:32:09.171420808 (547h3m9.042503843s) [upload=479693]: Got next segment. Inline: false Dec 28 10:32:09.171446587 (547h3m9.042529622s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:32:09.171451076 (547h3m9.042534121s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:32:09.415778928 (547h3m9.286861973s) [seg_pos={0 0}, upload=479693]: Begin upload segment... Dec 28 10:32:09.415823763 (547h3m9.286906798s) [seg_pos={0 0}, upload=479693]: Done begin upload segment. Dec 28 10:32:09.415852628 (547h3m9.286935673s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.415915609 (547h3m9.286998644s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416427983 (547h3m9.287511019s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416839006 (547h3m9.287922041s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416935741 (547h3m9.288018786s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417242896 (547h3m9.288325941s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417527407 (547h3m9.288610442s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417826456 (547h3m9.288909491s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417930104 (547h3m9.289013149s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418096461 (547h3m9.289179506s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418451897 (547h3m9.289534942s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419025139 (547h3m9.290108184s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419682581 (547h3m9.290765626s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.420365671 (547h3m9.291448706s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421011522 (547h3m9.292094557s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421697077 (547h3m9.292780122s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:11.133534367 (547h3m11.004617412s) [upload=479693]: Next returned nil segment Dec 28 10:32:11.133569233 (547h3m11.004652278s) [upload=479693]: Waiting for error group managing segments... Dec 28 10:32:11.421476958 (547h3m11.292559993s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.423163530 (547h3m11.294246575s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.428115299 (547h3m11.299198334s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.436758778 (547h3m11.307841823s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439394376 (547h3m11.310477411s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439861225 (547h3m11.31094427s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.442937283 (547h3m11.314020318s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.444228622 (547h3m11.315311667s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.452679474 (547h3m11.323762519s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.457542493 (547h3m11.328625538s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.708367176 (547h3m11.579450221s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.716026030 (547h3m11.587109075s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434215195 (547h3m12.30529824s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434324403 (547h3m12.305407438s) [seg_pos={0 0}, upload=479693]: Segment reached optimal threshold of 13 pieces. Dec 28 10:32:12.434515456 (547h3m12.305598501s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434521548 (547h3m12.305604593s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434710859 (547h3m12.305793894s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434716900 (547h3m12.305799945s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434797904 (547h3m12.305880949s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434813053 (547h3m12.305896088s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434845295 (547h3m12.30592833s) [seg_pos={0 0}, upload=479693]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:32:12.434957198 (547h3m12.306040233s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:32:12.434962488 (547h3m12.306045523s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:32:12.725143372 (547h3m12.596226407s) [upload=479693]: Done uploading segments... Dec 28 10:32:12.725163410 (547h3m12.596246445s) [upload=479693]: Upload finished. err: <nil> Duration: 5.929856097s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.X........X.....X..................X..............X.X.......] duration: 5.000778147s trace: https://jaeger.dev.storj.io/trace/57df8e96b91a6205 FAIL (2025-12-28 10:38:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000815609s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..X........X.....X..................X..............X.X......] duration: 3.467988517s trace: https://jaeger.dev.storj.io/trace/57df8e96b91a6205 FAIL (2025-12-28 10:37:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.468144865s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X...................X..XX.....X..X............XX......] duration: 6.198266809s trace: https://jaeger.dev.storj.io/trace/12fdeb4fc6be55f2 FAIL (2025-12-28 10:37:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A) context deadline exceeded ======LOGS====== Dec 28 10:32:09.171162516 (547h3m9.042245551s) [upload=479693]: Starting upload Dec 28 10:32:09.171211930 (547h3m9.042294965s) [upload=479693]: Done starting upload Dec 28 10:32:09.171389138 (547h3m9.042472173s) [upload=479693]: Uploading segments... Dec 28 10:32:09.171420808 (547h3m9.042503843s) [upload=479693]: Got next segment. Inline: false Dec 28 10:32:09.171446587 (547h3m9.042529622s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:32:09.171451076 (547h3m9.042534121s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:32:09.415778928 (547h3m9.286861973s) [seg_pos={0 0}, upload=479693]: Begin upload segment... Dec 28 10:32:09.415823763 (547h3m9.286906798s) [seg_pos={0 0}, upload=479693]: Done begin upload segment. Dec 28 10:32:09.415852628 (547h3m9.286935673s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.415915609 (547h3m9.286998644s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416427983 (547h3m9.287511019s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416839006 (547h3m9.287922041s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416935741 (547h3m9.288018786s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417242896 (547h3m9.288325941s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417527407 (547h3m9.288610442s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417826456 (547h3m9.288909491s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417930104 (547h3m9.289013149s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418096461 (547h3m9.289179506s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418451897 (547h3m9.289534942s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419025139 (547h3m9.290108184s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419682581 (547h3m9.290765626s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.420365671 (547h3m9.291448706s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421011522 (547h3m9.292094557s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421697077 (547h3m9.292780122s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:11.133534367 (547h3m11.004617412s) [upload=479693]: Next returned nil segment Dec 28 10:32:11.133569233 (547h3m11.004652278s) [upload=479693]: Waiting for error group managing segments... Dec 28 10:32:11.421476958 (547h3m11.292559993s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.423163530 (547h3m11.294246575s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.428115299 (547h3m11.299198334s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.436758778 (547h3m11.307841823s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439394376 (547h3m11.310477411s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439861225 (547h3m11.31094427s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.442937283 (547h3m11.314020318s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.444228622 (547h3m11.315311667s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.452679474 (547h3m11.323762519s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.457542493 (547h3m11.328625538s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.708367176 (547h3m11.579450221s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.716026030 (547h3m11.587109075s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434215195 (547h3m12.30529824s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434324403 (547h3m12.305407438s) [seg_pos={0 0}, upload=479693]: Segment reached optimal threshold of 13 pieces. Dec 28 10:32:12.434515456 (547h3m12.305598501s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434521548 (547h3m12.305604593s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434710859 (547h3m12.305793894s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434716900 (547h3m12.305799945s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434797904 (547h3m12.305880949s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434813053 (547h3m12.305896088s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434845295 (547h3m12.30592833s) [seg_pos={0 0}, upload=479693]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:32:12.434957198 (547h3m12.306040233s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:32:12.434962488 (547h3m12.306045523s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:32:12.725143372 (547h3m12.596226407s) [upload=479693]: Done uploading segments... Dec 28 10:32:12.725163410 (547h3m12.596246445s) [upload=479693]: Upload finished. err: <nil> Duration: 6.198294221s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)