Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 869969 reports (Page 11596 of 17400)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.857420864s trace: https://jaeger.dev.storj.io/trace/1f9f83d8cc633aae FAIL (2025-12-09 15:23:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:59:32.716918810 (95h30m32.588001855s) [upload=83259]: Starting upload Dec 9 14:59:32.716988723 (95h30m32.588071758s) [upload=83259]: Done starting upload Dec 9 14:59:32.717137536 (95h30m32.588220581s) [upload=83259]: Uploading segments... Dec 9 14:59:32.717286900 (95h30m32.588369935s) [upload=83259]: Got next segment. Inline: false Dec 9 14:59:32.717341534 (95h30m32.58842458s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:59:32.717347646 (95h30m32.588430691s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 14:59:32.975580773 (95h30m32.846663828s) [seg_pos={0 0}, upload=83259]: Begin upload segment... Dec 9 14:59:32.975640838 (95h30m32.846723883s) [seg_pos={0 0}, upload=83259]: Done begin upload segment. Dec 9 14:59:32.975718646 (95h30m32.846801701s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.975774743 (95h30m32.846857788s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976391548 (95h30m32.847474593s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976648978 (95h30m32.847732023s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977223793 (95h30m32.848306838s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977895982 (95h30m32.848979057s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978304240 (95h30m32.849387285s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978738136 (95h30m32.849821181s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978777981 (95h30m32.849861016s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979321667 (95h30m32.850404712s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979840495 (95h30m32.85092354s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979932580 (95h30m32.851015625s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980394961 (95h30m32.851477996s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980744797 (95h30m32.851827832s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.982809799 (95h30m32.853892844s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.983317917 (95h30m32.854400992s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:34.680921451 (95h30m34.552004486s) [upload=83259]: Next returned nil segment Dec 9 14:59:34.680943153 (95h30m34.552026198s) [upload=83259]: Waiting for error group managing segments... Dec 9 14:59:34.985677890 (95h30m34.856760925s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.986532547 (95h30m34.857615592s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.991365901 (95h30m34.862448936s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.996089796 (95h30m34.867172831s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.999561678 (95h30m34.870644713s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001337950 (95h30m34.872420985s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001503816 (95h30m34.872586881s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012234980 (95h30m34.883318025s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012262382 (95h30m34.883345437s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.015363096 (95h30m34.886446131s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.016246058 (95h30m34.887329103s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.076907590 (95h30m34.947990635s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085585104 (95h30m34.956668149s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085609681 (95h30m34.956692726s) [seg_pos={0 0}, upload=83259]: Segment reached optimal threshold of 13 pieces. Dec 9 14:59:35.085857443 (95h30m34.956940488s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.085865989 (95h30m34.956949034s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086179496 (95h30m34.957262581s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.086188963 (95h30m34.957271999s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086341084 (95h30m34.957424129s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection Dec 9 14:59:35.086348858 (95h30m34.957431903s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086370269 (95h30m34.957453315s) [seg_pos={0 0}, upload=83259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 14:59:35.086452486 (95h30m34.957535531s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 14:59:35.086459168 (95h30m34.957542203s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:04:20.015922420 (95h35m19.887005475s) [upload=83259]: Done uploading segments... Dec 9 15:04:20.015991150 (95h35m19.887074196s) [upload=83259]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.85744945s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [......................X.......................X....X........] duration: 5.093308521s trace: https://jaeger.dev.storj.io/trace/5e8e3ddb11d5e892 FAIL (2025-12-09 15:23:36 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:57908->45.38.24.2:443: read: connection reset by peer Duration: 5.093347125s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX.XX.....XXX.......X......X....X..X.X.X..X....X............] duration: 6.081395991s trace: https://jaeger.dev.storj.io/trace/54e1c42dcb9743d7 FAIL (2025-12-09 15:23:35 UTC) orbiter warning: context deadline exceeded Duration: 6.081447659s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.X...X.....X.X.XXXXXXX.X....X..X.XXXX.XX...X......X...X.XX..] duration: 5.001427815s trace: https://jaeger.dev.storj.io/trace/1458394c11d428ea FAIL (2025-12-09 15:23:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001453282s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X..............................X.........................] duration: 4.816743583s trace: https://jaeger.dev.storj.io/trace/1a72c3cee1ba60f FAIL (2025-12-09 15:23:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.816774892s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.720153266s trace: https://jaeger.dev.storj.io/trace/1f9f83d8cc633aae FAIL (2025-12-09 15:23:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:59:32.716918810 (95h30m32.588001855s) [upload=83259]: Starting upload Dec 9 14:59:32.716988723 (95h30m32.588071758s) [upload=83259]: Done starting upload Dec 9 14:59:32.717137536 (95h30m32.588220581s) [upload=83259]: Uploading segments... Dec 9 14:59:32.717286900 (95h30m32.588369935s) [upload=83259]: Got next segment. Inline: false Dec 9 14:59:32.717341534 (95h30m32.58842458s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:59:32.717347646 (95h30m32.588430691s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 14:59:32.975580773 (95h30m32.846663828s) [seg_pos={0 0}, upload=83259]: Begin upload segment... Dec 9 14:59:32.975640838 (95h30m32.846723883s) [seg_pos={0 0}, upload=83259]: Done begin upload segment. Dec 9 14:59:32.975718646 (95h30m32.846801701s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.975774743 (95h30m32.846857788s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976391548 (95h30m32.847474593s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976648978 (95h30m32.847732023s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977223793 (95h30m32.848306838s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977895982 (95h30m32.848979057s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978304240 (95h30m32.849387285s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978738136 (95h30m32.849821181s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978777981 (95h30m32.849861016s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979321667 (95h30m32.850404712s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979840495 (95h30m32.85092354s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979932580 (95h30m32.851015625s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980394961 (95h30m32.851477996s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980744797 (95h30m32.851827832s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.982809799 (95h30m32.853892844s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.983317917 (95h30m32.854400992s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:34.680921451 (95h30m34.552004486s) [upload=83259]: Next returned nil segment Dec 9 14:59:34.680943153 (95h30m34.552026198s) [upload=83259]: Waiting for error group managing segments... Dec 9 14:59:34.985677890 (95h30m34.856760925s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.986532547 (95h30m34.857615592s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.991365901 (95h30m34.862448936s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.996089796 (95h30m34.867172831s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.999561678 (95h30m34.870644713s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001337950 (95h30m34.872420985s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001503816 (95h30m34.872586881s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012234980 (95h30m34.883318025s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012262382 (95h30m34.883345437s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.015363096 (95h30m34.886446131s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.016246058 (95h30m34.887329103s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.076907590 (95h30m34.947990635s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085585104 (95h30m34.956668149s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085609681 (95h30m34.956692726s) [seg_pos={0 0}, upload=83259]: Segment reached optimal threshold of 13 pieces. Dec 9 14:59:35.085857443 (95h30m34.956940488s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.085865989 (95h30m34.956949034s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086179496 (95h30m34.957262581s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.086188963 (95h30m34.957271999s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086341084 (95h30m34.957424129s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection Dec 9 14:59:35.086348858 (95h30m34.957431903s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086370269 (95h30m34.957453315s) [seg_pos={0 0}, upload=83259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 14:59:35.086452486 (95h30m34.957535531s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 14:59:35.086459168 (95h30m34.957542203s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:04:20.015922420 (95h35m19.887005475s) [upload=83259]: Done uploading segments... Dec 9 15:04:20.015991150 (95h35m19.887074196s) [upload=83259]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.720182692s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [..X...X.....X.X.XXXXXXX.X....X..X.XXXX.XX...X......X...X.XX.] duration: 5.001608921s trace: https://jaeger.dev.storj.io/trace/1458394c11d428ea FAIL (2025-12-09 15:22:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001639949s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.82300177s trace: https://jaeger.dev.storj.io/trace/1f9f83d8cc633aae FAIL (2025-12-09 15:22:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:59:32.716918810 (95h30m32.588001855s) [upload=83259]: Starting upload Dec 9 14:59:32.716988723 (95h30m32.588071758s) [upload=83259]: Done starting upload Dec 9 14:59:32.717137536 (95h30m32.588220581s) [upload=83259]: Uploading segments... Dec 9 14:59:32.717286900 (95h30m32.588369935s) [upload=83259]: Got next segment. Inline: false Dec 9 14:59:32.717341534 (95h30m32.58842458s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:59:32.717347646 (95h30m32.588430691s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 14:59:32.975580773 (95h30m32.846663828s) [seg_pos={0 0}, upload=83259]: Begin upload segment... Dec 9 14:59:32.975640838 (95h30m32.846723883s) [seg_pos={0 0}, upload=83259]: Done begin upload segment. Dec 9 14:59:32.975718646 (95h30m32.846801701s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.975774743 (95h30m32.846857788s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976391548 (95h30m32.847474593s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976648978 (95h30m32.847732023s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977223793 (95h30m32.848306838s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977895982 (95h30m32.848979057s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978304240 (95h30m32.849387285s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978738136 (95h30m32.849821181s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978777981 (95h30m32.849861016s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979321667 (95h30m32.850404712s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979840495 (95h30m32.85092354s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979932580 (95h30m32.851015625s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980394961 (95h30m32.851477996s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980744797 (95h30m32.851827832s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.982809799 (95h30m32.853892844s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.983317917 (95h30m32.854400992s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:34.680921451 (95h30m34.552004486s) [upload=83259]: Next returned nil segment Dec 9 14:59:34.680943153 (95h30m34.552026198s) [upload=83259]: Waiting for error group managing segments... Dec 9 14:59:34.985677890 (95h30m34.856760925s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.986532547 (95h30m34.857615592s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.991365901 (95h30m34.862448936s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.996089796 (95h30m34.867172831s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.999561678 (95h30m34.870644713s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001337950 (95h30m34.872420985s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001503816 (95h30m34.872586881s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012234980 (95h30m34.883318025s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012262382 (95h30m34.883345437s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.015363096 (95h30m34.886446131s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.016246058 (95h30m34.887329103s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.076907590 (95h30m34.947990635s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085585104 (95h30m34.956668149s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085609681 (95h30m34.956692726s) [seg_pos={0 0}, upload=83259]: Segment reached optimal threshold of 13 pieces. Dec 9 14:59:35.085857443 (95h30m34.956940488s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.085865989 (95h30m34.956949034s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086179496 (95h30m34.957262581s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.086188963 (95h30m34.957271999s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086341084 (95h30m34.957424129s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection Dec 9 14:59:35.086348858 (95h30m34.957431903s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086370269 (95h30m34.957453315s) [seg_pos={0 0}, upload=83259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 14:59:35.086452486 (95h30m34.957535531s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 14:59:35.086459168 (95h30m34.957542203s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:04:20.015922420 (95h35m19.887005475s) [upload=83259]: Done uploading segments... Dec 9 15:04:20.015991150 (95h35m19.887074196s) [upload=83259]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.823023853s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.......................X....X.......] duration: 3.390912808s trace: https://jaeger.dev.storj.io/trace/5e8e3ddb11d5e892 FAIL (2025-12-09 15:22:37 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:57908->45.38.24.2:443: read: connection reset by peer Duration: 3.390946543s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XX.XX.....XXX.......X......X....X..X.X.X..X....X...........] duration: 6.459241161s trace: https://jaeger.dev.storj.io/trace/54e1c42dcb9743d7 FAIL (2025-12-09 15:22:35 UTC) orbiter warning: context deadline exceeded Duration: 6.459440149s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..............................X........................] duration: 5.051778283s trace: https://jaeger.dev.storj.io/trace/1a72c3cee1ba60f FAIL (2025-12-09 15:22:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.051811776s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.XX.XX.....XXX.......X......X....X..X.X.X..X....X..........] duration: 6.030233161s trace: https://jaeger.dev.storj.io/trace/54e1c42dcb9743d7 FAIL (2025-12-09 15:22:05 UTC) orbiter warning: context deadline exceeded Duration: 6.030281332s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..............................X.......................] duration: 4.789719282s trace: https://jaeger.dev.storj.io/trace/1a72c3cee1ba60f FAIL (2025-12-09 15:21:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.789821526s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [........................X.......................X....X......] duration: 3.646297052s trace: https://jaeger.dev.storj.io/trace/5e8e3ddb11d5e892 FAIL (2025-12-09 15:21:59 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:57908->45.38.24.2:443: read: connection reset by peer Duration: 3.646364701s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [X..X...X.....X.X.XXXXXXX.X....X..X.XXXX.XX...X......X...X.XX] duration: 5.001628579s trace: https://jaeger.dev.storj.io/trace/1458394c11d428ea FAIL (2025-12-09 15:21:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001653617s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.806440295s trace: https://jaeger.dev.storj.io/trace/1f9f83d8cc633aae FAIL (2025-12-09 15:21:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:59:32.716918810 (95h30m32.588001855s) [upload=83259]: Starting upload Dec 9 14:59:32.716988723 (95h30m32.588071758s) [upload=83259]: Done starting upload Dec 9 14:59:32.717137536 (95h30m32.588220581s) [upload=83259]: Uploading segments... Dec 9 14:59:32.717286900 (95h30m32.588369935s) [upload=83259]: Got next segment. Inline: false Dec 9 14:59:32.717341534 (95h30m32.58842458s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:59:32.717347646 (95h30m32.588430691s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 14:59:32.975580773 (95h30m32.846663828s) [seg_pos={0 0}, upload=83259]: Begin upload segment... Dec 9 14:59:32.975640838 (95h30m32.846723883s) [seg_pos={0 0}, upload=83259]: Done begin upload segment. Dec 9 14:59:32.975718646 (95h30m32.846801701s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.975774743 (95h30m32.846857788s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976391548 (95h30m32.847474593s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976648978 (95h30m32.847732023s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977223793 (95h30m32.848306838s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977895982 (95h30m32.848979057s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978304240 (95h30m32.849387285s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978738136 (95h30m32.849821181s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978777981 (95h30m32.849861016s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979321667 (95h30m32.850404712s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979840495 (95h30m32.85092354s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979932580 (95h30m32.851015625s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980394961 (95h30m32.851477996s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980744797 (95h30m32.851827832s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.982809799 (95h30m32.853892844s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.983317917 (95h30m32.854400992s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:34.680921451 (95h30m34.552004486s) [upload=83259]: Next returned nil segment Dec 9 14:59:34.680943153 (95h30m34.552026198s) [upload=83259]: Waiting for error group managing segments... Dec 9 14:59:34.985677890 (95h30m34.856760925s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.986532547 (95h30m34.857615592s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.991365901 (95h30m34.862448936s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.996089796 (95h30m34.867172831s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.999561678 (95h30m34.870644713s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001337950 (95h30m34.872420985s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001503816 (95h30m34.872586881s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012234980 (95h30m34.883318025s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012262382 (95h30m34.883345437s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.015363096 (95h30m34.886446131s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.016246058 (95h30m34.887329103s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.076907590 (95h30m34.947990635s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085585104 (95h30m34.956668149s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085609681 (95h30m34.956692726s) [seg_pos={0 0}, upload=83259]: Segment reached optimal threshold of 13 pieces. Dec 9 14:59:35.085857443 (95h30m34.956940488s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.085865989 (95h30m34.956949034s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086179496 (95h30m34.957262581s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.086188963 (95h30m34.957271999s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086341084 (95h30m34.957424129s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection Dec 9 14:59:35.086348858 (95h30m34.957431903s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086370269 (95h30m34.957453315s) [seg_pos={0 0}, upload=83259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 14:59:35.086452486 (95h30m34.957535531s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 14:59:35.086459168 (95h30m34.957542203s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:04:20.015922420 (95h35m19.887005475s) [upload=83259]: Done uploading segments... Dec 9 15:04:20.015991150 (95h35m19.887074196s) [upload=83259]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.806468007s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X..............................X......................] duration: 5.138281362s trace: https://jaeger.dev.storj.io/trace/1a72c3cee1ba60f FAIL (2025-12-09 15:21:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.138305729s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.XX.XX.....XXX.......X......X....X..X.X.X..X....X.........] duration: 8.820304095s trace: https://jaeger.dev.storj.io/trace/54e1c42dcb9743d7 FAIL (2025-12-09 15:21:27 UTC) orbiter warning: context deadline exceeded Duration: 8.820334564s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X.XX.XX.....XXX.......X......X....X..X.X.X..X....X........] duration: 12.29526521s trace: https://jaeger.dev.storj.io/trace/54e1c42dcb9743d7 FAIL (2025-12-09 15:21:03 UTC) orbiter warning: context deadline exceeded Duration: 12.295313762s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.X..X...X.....X.X.XXXXXXX.X....X..X.XXXX.XX...X......X...X.X] duration: 5.001240787s trace: https://jaeger.dev.storj.io/trace/638c1d53cd96134c FAIL (2025-12-09 15:20:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00139522s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.......................X....X.....] duration: 4.509057983s trace: https://jaeger.dev.storj.io/trace/5e8e3ddb11d5e892 FAIL (2025-12-09 15:20:51 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:57908->45.38.24.2:443: read: connection reset by peer Duration: 4.509165007s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.881147086s trace: https://jaeger.dev.storj.io/trace/1f9f83d8cc633aae FAIL (2025-12-09 15:20:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:59:32.716918810 (95h30m32.588001855s) [upload=83259]: Starting upload Dec 9 14:59:32.716988723 (95h30m32.588071758s) [upload=83259]: Done starting upload Dec 9 14:59:32.717137536 (95h30m32.588220581s) [upload=83259]: Uploading segments... Dec 9 14:59:32.717286900 (95h30m32.588369935s) [upload=83259]: Got next segment. Inline: false Dec 9 14:59:32.717341534 (95h30m32.58842458s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:59:32.717347646 (95h30m32.588430691s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 14:59:32.975580773 (95h30m32.846663828s) [seg_pos={0 0}, upload=83259]: Begin upload segment... Dec 9 14:59:32.975640838 (95h30m32.846723883s) [seg_pos={0 0}, upload=83259]: Done begin upload segment. Dec 9 14:59:32.975718646 (95h30m32.846801701s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.975774743 (95h30m32.846857788s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976391548 (95h30m32.847474593s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976648978 (95h30m32.847732023s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977223793 (95h30m32.848306838s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977895982 (95h30m32.848979057s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978304240 (95h30m32.849387285s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978738136 (95h30m32.849821181s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978777981 (95h30m32.849861016s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979321667 (95h30m32.850404712s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979840495 (95h30m32.85092354s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979932580 (95h30m32.851015625s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980394961 (95h30m32.851477996s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980744797 (95h30m32.851827832s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.982809799 (95h30m32.853892844s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.983317917 (95h30m32.854400992s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:34.680921451 (95h30m34.552004486s) [upload=83259]: Next returned nil segment Dec 9 14:59:34.680943153 (95h30m34.552026198s) [upload=83259]: Waiting for error group managing segments... Dec 9 14:59:34.985677890 (95h30m34.856760925s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.986532547 (95h30m34.857615592s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.991365901 (95h30m34.862448936s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.996089796 (95h30m34.867172831s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.999561678 (95h30m34.870644713s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001337950 (95h30m34.872420985s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001503816 (95h30m34.872586881s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012234980 (95h30m34.883318025s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012262382 (95h30m34.883345437s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.015363096 (95h30m34.886446131s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.016246058 (95h30m34.887329103s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.076907590 (95h30m34.947990635s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085585104 (95h30m34.956668149s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085609681 (95h30m34.956692726s) [seg_pos={0 0}, upload=83259]: Segment reached optimal threshold of 13 pieces. Dec 9 14:59:35.085857443 (95h30m34.956940488s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.085865989 (95h30m34.956949034s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086179496 (95h30m34.957262581s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.086188963 (95h30m34.957271999s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086341084 (95h30m34.957424129s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection Dec 9 14:59:35.086348858 (95h30m34.957431903s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086370269 (95h30m34.957453315s) [seg_pos={0 0}, upload=83259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 14:59:35.086452486 (95h30m34.957535531s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 14:59:35.086459168 (95h30m34.957542203s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:04:20.015922420 (95h35m19.887005475s) [upload=83259]: Done uploading segments... Dec 9 15:04:20.015991150 (95h35m19.887074196s) [upload=83259]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.881178656s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X..............................X.....................] duration: 5.005617313s trace: https://jaeger.dev.storj.io/trace/1a72c3cee1ba60f FAIL (2025-12-09 15:20:37 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.005646048s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.......................X....X....] duration: 5.479832456s trace: https://jaeger.dev.storj.io/trace/5e8e3ddb11d5e892 FAIL (2025-12-09 15:20:34 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:57908->45.38.24.2:443: read: connection reset by peer Duration: 5.479859257s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.025444791s trace: https://jaeger.dev.storj.io/trace/1f9f83d8cc633aae FAIL (2025-12-09 15:20:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:59:32.716918810 (95h30m32.588001855s) [upload=83259]: Starting upload Dec 9 14:59:32.716988723 (95h30m32.588071758s) [upload=83259]: Done starting upload Dec 9 14:59:32.717137536 (95h30m32.588220581s) [upload=83259]: Uploading segments... Dec 9 14:59:32.717286900 (95h30m32.588369935s) [upload=83259]: Got next segment. Inline: false Dec 9 14:59:32.717341534 (95h30m32.58842458s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:59:32.717347646 (95h30m32.588430691s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 14:59:32.975580773 (95h30m32.846663828s) [seg_pos={0 0}, upload=83259]: Begin upload segment... Dec 9 14:59:32.975640838 (95h30m32.846723883s) [seg_pos={0 0}, upload=83259]: Done begin upload segment. Dec 9 14:59:32.975718646 (95h30m32.846801701s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.975774743 (95h30m32.846857788s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976391548 (95h30m32.847474593s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976648978 (95h30m32.847732023s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977223793 (95h30m32.848306838s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977895982 (95h30m32.848979057s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978304240 (95h30m32.849387285s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978738136 (95h30m32.849821181s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978777981 (95h30m32.849861016s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979321667 (95h30m32.850404712s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979840495 (95h30m32.85092354s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979932580 (95h30m32.851015625s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980394961 (95h30m32.851477996s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980744797 (95h30m32.851827832s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.982809799 (95h30m32.853892844s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.983317917 (95h30m32.854400992s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:34.680921451 (95h30m34.552004486s) [upload=83259]: Next returned nil segment Dec 9 14:59:34.680943153 (95h30m34.552026198s) [upload=83259]: Waiting for error group managing segments... Dec 9 14:59:34.985677890 (95h30m34.856760925s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.986532547 (95h30m34.857615592s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.991365901 (95h30m34.862448936s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.996089796 (95h30m34.867172831s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.999561678 (95h30m34.870644713s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001337950 (95h30m34.872420985s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001503816 (95h30m34.872586881s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012234980 (95h30m34.883318025s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012262382 (95h30m34.883345437s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.015363096 (95h30m34.886446131s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.016246058 (95h30m34.887329103s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.076907590 (95h30m34.947990635s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085585104 (95h30m34.956668149s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085609681 (95h30m34.956692726s) [seg_pos={0 0}, upload=83259]: Segment reached optimal threshold of 13 pieces. Dec 9 14:59:35.085857443 (95h30m34.956940488s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.085865989 (95h30m34.956949034s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086179496 (95h30m34.957262581s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.086188963 (95h30m34.957271999s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086341084 (95h30m34.957424129s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection Dec 9 14:59:35.086348858 (95h30m34.957431903s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086370269 (95h30m34.957453315s) [seg_pos={0 0}, upload=83259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 14:59:35.086452486 (95h30m34.957535531s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 14:59:35.086459168 (95h30m34.957542203s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:04:20.015922420 (95h35m19.887005475s) [upload=83259]: Done uploading segments... Dec 9 15:04:20.015991150 (95h35m19.887074196s) [upload=83259]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.02547579s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X..............................X....................] duration: 4.83903796s trace: https://jaeger.dev.storj.io/trace/1a72c3cee1ba60f FAIL (2025-12-09 15:20:12 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.839091412s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [X.X..X...X.....X.X.XXXXXXX.X....X..X.XXXX.XX...X......X...X.] duration: 5.001049107s trace: https://jaeger.dev.storj.io/trace/5f57d23532292fe2 FAIL (2025-12-09 15:20:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001081998s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X.XX.XX.....XXX.......X......X....X..X.X.X..X....X.......] duration: 19.656214632s trace: https://jaeger.dev.storj.io/trace/54e1c42dcb9743d7 FAIL (2025-12-09 15:19:57 UTC) orbiter warning: context deadline exceeded Duration: 19.656337227s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X..............................X...................] duration: 5.157865246s trace: https://jaeger.dev.storj.io/trace/1a72c3cee1ba60f FAIL (2025-12-09 15:19:21 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.157891297s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.......................X....X...] duration: 4.080408288s trace: https://jaeger.dev.storj.io/trace/5e8e3ddb11d5e892 FAIL (2025-12-09 15:19:19 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:57908->45.38.24.2:443: read: connection reset by peer Duration: 4.080439567s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.835400803s trace: https://jaeger.dev.storj.io/trace/1f9f83d8cc633aae FAIL (2025-12-09 15:19:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:59:32.716918810 (95h30m32.588001855s) [upload=83259]: Starting upload Dec 9 14:59:32.716988723 (95h30m32.588071758s) [upload=83259]: Done starting upload Dec 9 14:59:32.717137536 (95h30m32.588220581s) [upload=83259]: Uploading segments... Dec 9 14:59:32.717286900 (95h30m32.588369935s) [upload=83259]: Got next segment. Inline: false Dec 9 14:59:32.717341534 (95h30m32.58842458s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:59:32.717347646 (95h30m32.588430691s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 14:59:32.975580773 (95h30m32.846663828s) [seg_pos={0 0}, upload=83259]: Begin upload segment... Dec 9 14:59:32.975640838 (95h30m32.846723883s) [seg_pos={0 0}, upload=83259]: Done begin upload segment. Dec 9 14:59:32.975718646 (95h30m32.846801701s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.975774743 (95h30m32.846857788s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976391548 (95h30m32.847474593s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976648978 (95h30m32.847732023s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977223793 (95h30m32.848306838s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977895982 (95h30m32.848979057s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978304240 (95h30m32.849387285s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978738136 (95h30m32.849821181s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978777981 (95h30m32.849861016s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979321667 (95h30m32.850404712s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979840495 (95h30m32.85092354s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979932580 (95h30m32.851015625s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980394961 (95h30m32.851477996s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980744797 (95h30m32.851827832s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.982809799 (95h30m32.853892844s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.983317917 (95h30m32.854400992s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:34.680921451 (95h30m34.552004486s) [upload=83259]: Next returned nil segment Dec 9 14:59:34.680943153 (95h30m34.552026198s) [upload=83259]: Waiting for error group managing segments... Dec 9 14:59:34.985677890 (95h30m34.856760925s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.986532547 (95h30m34.857615592s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.991365901 (95h30m34.862448936s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.996089796 (95h30m34.867172831s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.999561678 (95h30m34.870644713s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001337950 (95h30m34.872420985s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001503816 (95h30m34.872586881s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012234980 (95h30m34.883318025s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012262382 (95h30m34.883345437s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.015363096 (95h30m34.886446131s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.016246058 (95h30m34.887329103s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.076907590 (95h30m34.947990635s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085585104 (95h30m34.956668149s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085609681 (95h30m34.956692726s) [seg_pos={0 0}, upload=83259]: Segment reached optimal threshold of 13 pieces. Dec 9 14:59:35.085857443 (95h30m34.956940488s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.085865989 (95h30m34.956949034s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086179496 (95h30m34.957262581s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.086188963 (95h30m34.957271999s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086341084 (95h30m34.957424129s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection Dec 9 14:59:35.086348858 (95h30m34.957431903s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086370269 (95h30m34.957453315s) [seg_pos={0 0}, upload=83259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 14:59:35.086452486 (95h30m34.957535531s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 14:59:35.086459168 (95h30m34.957542203s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:04:20.015922420 (95h35m19.887005475s) [upload=83259]: Done uploading segments... Dec 9 15:04:20.015991150 (95h35m19.887074196s) [upload=83259]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.835429688s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [XX.X..X...X.....X.X.XXXXXXX.X....X..X.XXXX.XX...X......X...X] duration: 5.001428242s trace: https://jaeger.dev.storj.io/trace/5f57d23532292fe2 FAIL (2025-12-09 15:19:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00145439s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X..............................X..................] duration: 4.987756746s trace: https://jaeger.dev.storj.io/trace/1a72c3cee1ba60f FAIL (2025-12-09 15:18:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.987787464s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X.XX.XX.....XXX.......X......X....X..X.X.X..X....X......] duration: 9.890588969s trace: https://jaeger.dev.storj.io/trace/54e1c42dcb9743d7 FAIL (2025-12-09 15:18:46 UTC) orbiter warning: context deadline exceeded Duration: 9.890618465s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.......................................X...................] duration: 14.956978725s trace: https://jaeger.dev.storj.io/trace/1f9f83d8cc633aae FAIL (2025-12-09 15:18:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:59:32.716918810 (95h30m32.588001855s) [upload=83259]: Starting upload Dec 9 14:59:32.716988723 (95h30m32.588071758s) [upload=83259]: Done starting upload Dec 9 14:59:32.717137536 (95h30m32.588220581s) [upload=83259]: Uploading segments... Dec 9 14:59:32.717286900 (95h30m32.588369935s) [upload=83259]: Got next segment. Inline: false Dec 9 14:59:32.717341534 (95h30m32.58842458s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:59:32.717347646 (95h30m32.588430691s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 14:59:32.975580773 (95h30m32.846663828s) [seg_pos={0 0}, upload=83259]: Begin upload segment... Dec 9 14:59:32.975640838 (95h30m32.846723883s) [seg_pos={0 0}, upload=83259]: Done begin upload segment. Dec 9 14:59:32.975718646 (95h30m32.846801701s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.975774743 (95h30m32.846857788s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976391548 (95h30m32.847474593s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976648978 (95h30m32.847732023s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977223793 (95h30m32.848306838s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977895982 (95h30m32.848979057s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978304240 (95h30m32.849387285s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978738136 (95h30m32.849821181s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978777981 (95h30m32.849861016s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979321667 (95h30m32.850404712s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979840495 (95h30m32.85092354s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979932580 (95h30m32.851015625s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980394961 (95h30m32.851477996s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980744797 (95h30m32.851827832s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.982809799 (95h30m32.853892844s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.983317917 (95h30m32.854400992s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:34.680921451 (95h30m34.552004486s) [upload=83259]: Next returned nil segment Dec 9 14:59:34.680943153 (95h30m34.552026198s) [upload=83259]: Waiting for error group managing segments... Dec 9 14:59:34.985677890 (95h30m34.856760925s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.986532547 (95h30m34.857615592s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.991365901 (95h30m34.862448936s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.996089796 (95h30m34.867172831s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.999561678 (95h30m34.870644713s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001337950 (95h30m34.872420985s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001503816 (95h30m34.872586881s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012234980 (95h30m34.883318025s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012262382 (95h30m34.883345437s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.015363096 (95h30m34.886446131s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.016246058 (95h30m34.887329103s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.076907590 (95h30m34.947990635s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085585104 (95h30m34.956668149s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085609681 (95h30m34.956692726s) [seg_pos={0 0}, upload=83259]: Segment reached optimal threshold of 13 pieces. Dec 9 14:59:35.085857443 (95h30m34.956940488s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.085865989 (95h30m34.956949034s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086179496 (95h30m34.957262581s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.086188963 (95h30m34.957271999s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086341084 (95h30m34.957424129s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection Dec 9 14:59:35.086348858 (95h30m34.957431903s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086370269 (95h30m34.957453315s) [seg_pos={0 0}, upload=83259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 14:59:35.086452486 (95h30m34.957535531s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 14:59:35.086459168 (95h30m34.957542203s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:04:20.015922420 (95h35m19.887005475s) [upload=83259]: Done uploading segments... Dec 9 15:04:20.015991150 (95h35m19.887074196s) [upload=83259]: Upload finished. err: metaclient: context deadline exceeded Duration: 14.957064538s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [............................X.......................X....X..] duration: 7.463376888s trace: https://jaeger.dev.storj.io/trace/5e8e3ddb11d5e892 FAIL (2025-12-09 15:18:40 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:57908->45.38.24.2:443: read: connection reset by peer Duration: 7.463464295s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.....X.XX.XX.....XXX.......X......X....X..X.X.X..X....X.....] duration: 23.461978129s trace: https://jaeger.dev.storj.io/trace/54e1c42dcb9743d7 FAIL (2025-12-09 15:18:32 UTC) orbiter warning: context deadline exceeded Duration: 23.462007585s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [XXX.X..X...X.....X.X.XXXXXXX.X....X..X.XXXX.XX...X......X...] duration: 4.988194185s trace: https://jaeger.dev.storj.io/trace/3736c72860d6faa1 FAIL (2025-12-09 15:18:30 UTC) orbiter warning: context deadline exceeded Duration: 4.988220215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X..............................X.................] duration: 4.976687644s trace: https://jaeger.dev.storj.io/trace/1a72c3cee1ba60f FAIL (2025-12-09 15:18:10 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.976713524s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.......................................X..................] duration: 5.770601258s trace: https://jaeger.dev.storj.io/trace/1f9f83d8cc633aae FAIL (2025-12-09 15:18:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:59:32.716918810 (95h30m32.588001855s) [upload=83259]: Starting upload Dec 9 14:59:32.716988723 (95h30m32.588071758s) [upload=83259]: Done starting upload Dec 9 14:59:32.717137536 (95h30m32.588220581s) [upload=83259]: Uploading segments... Dec 9 14:59:32.717286900 (95h30m32.588369935s) [upload=83259]: Got next segment. Inline: false Dec 9 14:59:32.717341534 (95h30m32.58842458s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:59:32.717347646 (95h30m32.588430691s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 14:59:32.975580773 (95h30m32.846663828s) [seg_pos={0 0}, upload=83259]: Begin upload segment... Dec 9 14:59:32.975640838 (95h30m32.846723883s) [seg_pos={0 0}, upload=83259]: Done begin upload segment. Dec 9 14:59:32.975718646 (95h30m32.846801701s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.975774743 (95h30m32.846857788s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976391548 (95h30m32.847474593s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976648978 (95h30m32.847732023s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977223793 (95h30m32.848306838s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977895982 (95h30m32.848979057s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978304240 (95h30m32.849387285s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978738136 (95h30m32.849821181s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978777981 (95h30m32.849861016s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979321667 (95h30m32.850404712s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979840495 (95h30m32.85092354s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979932580 (95h30m32.851015625s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980394961 (95h30m32.851477996s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980744797 (95h30m32.851827832s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.982809799 (95h30m32.853892844s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.983317917 (95h30m32.854400992s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:34.680921451 (95h30m34.552004486s) [upload=83259]: Next returned nil segment Dec 9 14:59:34.680943153 (95h30m34.552026198s) [upload=83259]: Waiting for error group managing segments... Dec 9 14:59:34.985677890 (95h30m34.856760925s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.986532547 (95h30m34.857615592s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.991365901 (95h30m34.862448936s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.996089796 (95h30m34.867172831s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.999561678 (95h30m34.870644713s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001337950 (95h30m34.872420985s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001503816 (95h30m34.872586881s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012234980 (95h30m34.883318025s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012262382 (95h30m34.883345437s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.015363096 (95h30m34.886446131s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.016246058 (95h30m34.887329103s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.076907590 (95h30m34.947990635s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085585104 (95h30m34.956668149s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085609681 (95h30m34.956692726s) [seg_pos={0 0}, upload=83259]: Segment reached optimal threshold of 13 pieces. Dec 9 14:59:35.085857443 (95h30m34.956940488s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.085865989 (95h30m34.956949034s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086179496 (95h30m34.957262581s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.086188963 (95h30m34.957271999s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086341084 (95h30m34.957424129s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection Dec 9 14:59:35.086348858 (95h30m34.957431903s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086370269 (95h30m34.957453315s) [seg_pos={0 0}, upload=83259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 14:59:35.086452486 (95h30m34.957535531s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 14:59:35.086459168 (95h30m34.957542203s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:04:20.015922420 (95h35m19.887005475s) [upload=83259]: Done uploading segments... Dec 9 15:04:20.015991150 (95h35m19.887074196s) [upload=83259]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.77062803s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.......................X....X.] duration: 5.421730996s trace: https://jaeger.dev.storj.io/trace/5e8e3ddb11d5e892 FAIL (2025-12-09 15:18:04 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:57908->45.38.24.2:443: read: connection reset by peer Duration: 5.421759601s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X..............................X................] duration: 4.79724022s trace: https://jaeger.dev.storj.io/trace/1a72c3cee1ba60f FAIL (2025-12-09 15:17:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.797268342s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [.XXX.X..X...X.....X.X.XXXXXXX.X....X..X.XXXX.XX...X......X..] duration: 5.001002746s trace: https://jaeger.dev.storj.io/trace/3736c72860d6faa1 FAIL (2025-12-09 15:17:55 UTC) orbiter warning: context deadline exceeded Duration: 5.001030119s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......X.XX.XX.....XXX.......X......X....X..X.X.X..X....X....] duration: 29.839243164s trace: https://jaeger.dev.storj.io/trace/54e1c42dcb9743d7 FAIL (2025-12-09 15:17:45 UTC) orbiter warning: context deadline exceeded Duration: 29.839267611s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.......................................X.................] duration: 5.911447005s trace: https://jaeger.dev.storj.io/trace/1f9f83d8cc633aae FAIL (2025-12-09 15:17:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:59:32.716918810 (95h30m32.588001855s) [upload=83259]: Starting upload Dec 9 14:59:32.716988723 (95h30m32.588071758s) [upload=83259]: Done starting upload Dec 9 14:59:32.717137536 (95h30m32.588220581s) [upload=83259]: Uploading segments... Dec 9 14:59:32.717286900 (95h30m32.588369935s) [upload=83259]: Got next segment. Inline: false Dec 9 14:59:32.717341534 (95h30m32.58842458s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:59:32.717347646 (95h30m32.588430691s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 14:59:32.975580773 (95h30m32.846663828s) [seg_pos={0 0}, upload=83259]: Begin upload segment... Dec 9 14:59:32.975640838 (95h30m32.846723883s) [seg_pos={0 0}, upload=83259]: Done begin upload segment. Dec 9 14:59:32.975718646 (95h30m32.846801701s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.975774743 (95h30m32.846857788s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976391548 (95h30m32.847474593s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976648978 (95h30m32.847732023s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977223793 (95h30m32.848306838s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977895982 (95h30m32.848979057s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978304240 (95h30m32.849387285s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978738136 (95h30m32.849821181s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978777981 (95h30m32.849861016s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979321667 (95h30m32.850404712s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979840495 (95h30m32.85092354s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979932580 (95h30m32.851015625s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980394961 (95h30m32.851477996s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980744797 (95h30m32.851827832s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.982809799 (95h30m32.853892844s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.983317917 (95h30m32.854400992s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:34.680921451 (95h30m34.552004486s) [upload=83259]: Next returned nil segment Dec 9 14:59:34.680943153 (95h30m34.552026198s) [upload=83259]: Waiting for error group managing segments... Dec 9 14:59:34.985677890 (95h30m34.856760925s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.986532547 (95h30m34.857615592s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.991365901 (95h30m34.862448936s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.996089796 (95h30m34.867172831s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.999561678 (95h30m34.870644713s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001337950 (95h30m34.872420985s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001503816 (95h30m34.872586881s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012234980 (95h30m34.883318025s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012262382 (95h30m34.883345437s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.015363096 (95h30m34.886446131s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.016246058 (95h30m34.887329103s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.076907590 (95h30m34.947990635s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085585104 (95h30m34.956668149s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085609681 (95h30m34.956692726s) [seg_pos={0 0}, upload=83259]: Segment reached optimal threshold of 13 pieces. Dec 9 14:59:35.085857443 (95h30m34.956940488s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.085865989 (95h30m34.956949034s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086179496 (95h30m34.957262581s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.086188963 (95h30m34.957271999s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086341084 (95h30m34.957424129s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection Dec 9 14:59:35.086348858 (95h30m34.957431903s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086370269 (95h30m34.957453315s) [seg_pos={0 0}, upload=83259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 14:59:35.086452486 (95h30m34.957535531s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 14:59:35.086459168 (95h30m34.957542203s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:04:20.015922420 (95h35m19.887005475s) [upload=83259]: Done uploading segments... Dec 9 15:04:20.015991150 (95h35m19.887074196s) [upload=83259]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.911560009s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X..............................X...............] duration: 4.811129849s trace: https://jaeger.dev.storj.io/trace/1a72c3cee1ba60f FAIL (2025-12-09 15:17:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.811159584s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [..XXX.X..X...X.....X.X.XXXXXXX.X....X..X.XXXX.XX...X......X.] duration: 4.741629065s trace: https://jaeger.dev.storj.io/trace/3736c72860d6faa1 FAIL (2025-12-09 15:17:07 UTC) orbiter warning: context deadline exceeded Duration: 4.741787427s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.......X.XX.XX.....XXX.......X......X....X..X.X.X..X....X...] duration: 15.366838451s trace: https://jaeger.dev.storj.io/trace/54e1c42dcb9743d7 FAIL (2025-12-09 15:17:06 UTC) orbiter warning: context deadline exceeded Duration: 15.366875512s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.......................X....X] duration: 2.748556633s trace: https://jaeger.dev.storj.io/trace/5e8e3ddb11d5e892 FAIL (2025-12-09 15:16:59 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:57908->45.38.24.2:443: read: connection reset by peer Duration: 2.748586439s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.......................................X................] duration: 5.828102556s trace: https://jaeger.dev.storj.io/trace/1f9f83d8cc633aae FAIL (2025-12-09 15:16:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:59:32.716918810 (95h30m32.588001855s) [upload=83259]: Starting upload Dec 9 14:59:32.716988723 (95h30m32.588071758s) [upload=83259]: Done starting upload Dec 9 14:59:32.717137536 (95h30m32.588220581s) [upload=83259]: Uploading segments... Dec 9 14:59:32.717286900 (95h30m32.588369935s) [upload=83259]: Got next segment. Inline: false Dec 9 14:59:32.717341534 (95h30m32.58842458s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 14:59:32.717347646 (95h30m32.588430691s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 14:59:32.975580773 (95h30m32.846663828s) [seg_pos={0 0}, upload=83259]: Begin upload segment... Dec 9 14:59:32.975640838 (95h30m32.846723883s) [seg_pos={0 0}, upload=83259]: Done begin upload segment. Dec 9 14:59:32.975718646 (95h30m32.846801701s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.975774743 (95h30m32.846857788s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976391548 (95h30m32.847474593s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.976648978 (95h30m32.847732023s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977223793 (95h30m32.848306838s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.977895982 (95h30m32.848979057s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978304240 (95h30m32.849387285s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978738136 (95h30m32.849821181s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.978777981 (95h30m32.849861016s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979321667 (95h30m32.850404712s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979840495 (95h30m32.85092354s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.979932580 (95h30m32.851015625s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980394961 (95h30m32.851477996s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.980744797 (95h30m32.851827832s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.982809799 (95h30m32.853892844s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:32.983317917 (95h30m32.854400992s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Uploading piece... Dec 9 14:59:34.680921451 (95h30m34.552004486s) [upload=83259]: Next returned nil segment Dec 9 14:59:34.680943153 (95h30m34.552026198s) [upload=83259]: Waiting for error group managing segments... Dec 9 14:59:34.985677890 (95h30m34.856760925s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.986532547 (95h30m34.857615592s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.991365901 (95h30m34.862448936s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.996089796 (95h30m34.867172831s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:34.999561678 (95h30m34.870644713s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001337950 (95h30m34.872420985s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.001503816 (95h30m34.872586881s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012234980 (95h30m34.883318025s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.012262382 (95h30m34.883345437s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.015363096 (95h30m34.886446131s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.016246058 (95h30m34.887329103s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.076907590 (95h30m34.947990635s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085585104 (95h30m34.956668149s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil> Dec 9 14:59:35.085609681 (95h30m34.956692726s) [seg_pos={0 0}, upload=83259]: Segment reached optimal threshold of 13 pieces. Dec 9 14:59:35.085857443 (95h30m34.956940488s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.085865989 (95h30m34.956949034s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086179496 (95h30m34.957262581s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 9 14:59:35.086188963 (95h30m34.957271999s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086341084 (95h30m34.957424129s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection Dec 9 14:59:35.086348858 (95h30m34.957431903s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold Dec 9 14:59:35.086370269 (95h30m34.957453315s) [seg_pos={0 0}, upload=83259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 14:59:35.086452486 (95h30m34.957535531s) [upload=83259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 14:59:35.086459168 (95h30m34.957542203s) [upload=83259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:04:20.015922420 (95h35m19.887005475s) [upload=83259]: Done uploading segments... Dec 9 15:04:20.015991150 (95h35m19.887074196s) [upload=83259]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.82813544s core uplink us-select-1

1
...
11594
11595
11596
11597
11598
...
17400