Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX] duration: 5.001644041s trace: https://jaeger.dev.storj.io/trace/4c76ec3bafbf9883 FAIL (2025-12-10 14:09:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001667995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 15.241483985s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:09:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 15.241513311s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration: 5.00118812s trace: https://jaeger.dev.storj.io/trace/2fdcaeabe7c51991 FAIL (2025-12-10 14:09:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001214902s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.......................X..] duration: 18.7833785s trace: https://jaeger.dev.storj.io/trace/6e0208de34c7a7de FAIL (2025-12-10 14:09:12 UTC) orbiter warning: context deadline exceeded Duration: 18.783408096s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X.X....X......X.X....X.XXXX....X.XX...XX.X...X...X...X....] duration: 21.799262413s trace: https://jaeger.dev.storj.io/trace/48574c3a09eee4a9 FAIL (2025-12-10 14:08:59 UTC) orbiter warning: context deadline exceeded Duration: 21.79928804s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.......................X.] duration: 16.735753862s trace: https://jaeger.dev.storj.io/trace/6e0208de34c7a7de FAIL (2025-12-10 14:08:43 UTC) orbiter warning: context deadline exceeded Duration: 16.735770695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 13.466067638s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:08:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 13.466097275s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.......................X] duration: 1m0.001742431s trace: https://jaeger.dev.storj.io/trace/6e0208de34c7a7de FAIL (2025-12-10 14:08:26 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001755285s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X..X.X....X......X.X....X.XXXX....X.XX...XX.X...X...X...X...] duration: 20.107527147s trace: https://jaeger.dev.storj.io/trace/48574c3a09eee4a9 FAIL (2025-12-10 14:08:25 UTC) orbiter warning: context deadline exceeded Duration: 20.107577113s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 5.001326782s trace: https://jaeger.dev.storj.io/trace/433b620fad7192f6 FAIL (2025-12-10 14:08:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001347842s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 9.332216549s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:08:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 9.332244691s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 5.001384213s trace: https://jaeger.dev.storj.io/trace/43072d336c63dd64 FAIL (2025-12-10 14:07:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001413709s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X..X.X....X......X.X....X.XXXX....X.XX...XX.X...X...X...X..] duration: 23.918777398s trace: https://jaeger.dev.storj.io/trace/48574c3a09eee4a9 FAIL (2025-12-10 14:07:53 UTC) orbiter warning: context deadline exceeded Duration: 23.91883118s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 18.695938878s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:07:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 18.69597181s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 54.967257671s trace: https://jaeger.dev.storj.io/trace/44487f6e1b05dee9 FAIL (2025-12-10 14:07:26 UTC) orbiter warning: context deadline exceeded Duration: 54.967287959s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 22.784037043s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:07:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 22.78406643s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X..X.X....X......X.X....X.XXXX....X.XX...XX.X...X...X...X.] duration: 16.445149752s trace: https://jaeger.dev.storj.io/trace/48574c3a09eee4a9 FAIL (2025-12-10 14:07:02 UTC) orbiter warning: context deadline exceeded Duration: 16.445166433s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001354477s trace: https://jaeger.dev.storj.io/trace/43072d336c63dd64 FAIL (2025-12-10 14:06:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001378743s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...X..X.X....X......X.X....X.XXXX....X.XX...XX.X...X...X...X] duration: 30.001457065s trace: https://jaeger.dev.storj.io/trace/48574c3a09eee4a9 FAIL (2025-12-10 14:06:45 UTC) orbiter warning: context deadline exceeded Duration: 30.001482183s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001899923s trace: https://jaeger.dev.storj.io/trace/573ffd9815afb0a FAIL (2025-12-10 14:06:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00192977s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 35.00057381s trace: https://jaeger.dev.storj.io/trace/44487f6e1b05dee9 FAIL (2025-12-10 14:06:26 UTC) orbiter warning: context deadline exceeded Duration: 35.000625159s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....X..X.X....X......X.X....X.XXXX....X.XX...XX.X...X...X...] duration: 23.384486009s trace: https://jaeger.dev.storj.io/trace/20eeffc5dec891c7 FAIL (2025-12-10 14:06:03 UTC) orbiter warning: context deadline exceeded Duration: 23.38450752s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 12.878995061s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:05:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 12.879027703s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001649546s trace: https://jaeger.dev.storj.io/trace/29c86cf3ca9acd1 FAIL (2025-12-10 14:05:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001679773s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 24.474414581s trace: https://jaeger.dev.storj.io/trace/44487f6e1b05dee9 FAIL (2025-12-10 14:05:14 UTC) orbiter warning: context deadline exceeded Duration: 24.47443968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 8.938309159s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:05:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 8.938336291s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.....X..X.X....X......X.X....X.XXXX....X.XX...XX.X...X...X..] duration: 28.611091768s trace: https://jaeger.dev.storj.io/trace/20eeffc5dec891c7 FAIL (2025-12-10 14:05:07 UTC) orbiter warning: context deadline exceeded Duration: 28.611115424s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [......X..X.X....X......X.X....X.XXXX....X.XX...XX.X...X...X.] duration: 30.002548876s trace: https://jaeger.dev.storj.io/trace/20eeffc5dec891c7 FAIL (2025-12-10 14:04:38 UTC) orbiter warning: context deadline exceeded Duration: 30.002576889s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 22.043744698s trace: https://jaeger.dev.storj.io/trace/44487f6e1b05dee9 FAIL (2025-12-10 14:04:36 UTC) orbiter warning: context deadline exceeded Duration: 22.043763424s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001541574s trace: https://jaeger.dev.storj.io/trace/2dc864b0c8cf4712 FAIL (2025-12-10 14:04:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001567524s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 12.743986379s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:04:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 12.744018821s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001840932s trace: https://jaeger.dev.storj.io/trace/70a59ed5dad940b FAIL (2025-12-10 14:04:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001867552s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.......X..X.X....X......X.X....X.XXXX....X.XX...XX.X...X...X] duration: 30.001725345s trace: https://jaeger.dev.storj.io/trace/20eeffc5dec891c7 FAIL (2025-12-10 14:03:23 UTC) orbiter warning: context deadline exceeded Duration: 30.0017493s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.002683788s trace: https://jaeger.dev.storj.io/trace/78a72d28bfc122f5 FAIL (2025-12-10 14:03:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00271086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 9.214948023s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:03:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 9.214983711s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 16.419245348s trace: https://jaeger.dev.storj.io/trace/44487f6e1b05dee9 FAIL (2025-12-10 14:03:16 UTC) orbiter warning: context deadline exceeded Duration: 16.419269545s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [........X..X.X....X......X.X....X.XXXX....X.XX...XX.X...X...] duration: 26.419270357s trace: https://jaeger.dev.storj.io/trace/6ee38a3990af6f6 FAIL (2025-12-10 14:02:20 UTC) orbiter warning: context deadline exceeded Duration: 26.419294643s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001044278s trace: https://jaeger.dev.storj.io/trace/7a79d4fbaa50c3e1 FAIL (2025-12-10 14:02:15 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001058114s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 10.155056262s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:02:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 10.155179155s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 21.874969682s trace: https://jaeger.dev.storj.io/trace/44487f6e1b05dee9 FAIL (2025-12-10 14:02:10 UTC) orbiter warning: context deadline exceeded Duration: 21.874985973s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 9.718922887s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:01:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 9.718954837s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 21.804539911s trace: https://jaeger.dev.storj.io/trace/44487f6e1b05dee9 FAIL (2025-12-10 14:01:34 UTC) orbiter warning: context deadline exceeded Duration: 21.804579868s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.........X..X.X....X......X.X....X.XXXX....X.XX...XX.X...X..] duration: 28.654942073s trace: https://jaeger.dev.storj.io/trace/6ee38a3990af6f6 FAIL (2025-12-10 14:01:30 UTC) orbiter warning: context deadline exceeded Duration: 28.654962101s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 16.758553604s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:01:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 16.758583672s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXX.XXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001684105s trace: https://jaeger.dev.storj.io/trace/2725e40c541843f5 FAIL (2025-12-10 14:01:04 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001700207s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 17.88113301s trace: https://jaeger.dev.storj.io/trace/44487f6e1b05dee9 FAIL (2025-12-10 14:00:30 UTC) orbiter warning: context deadline exceeded Duration: 17.881162086s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001645782s trace: https://jaeger.dev.storj.io/trace/319aaa1bfb18af84 FAIL (2025-12-10 14:00:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001678634s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..........X..X.X....X......X.X....X.XXXX....X.XX...XX.X...X.] duration: 17.35437887s trace: https://jaeger.dev.storj.io/trace/6ee38a3990af6f6 FAIL (2025-12-10 14:00:09 UTC) orbiter warning: context deadline exceeded Duration: 17.354409921s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 14.538297063s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:00:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 14.538327581s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000758316s trace: https://jaeger.dev.storj.io/trace/2144dedf328535cf FAIL (2025-12-10 13:59:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000865631s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)