Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...X.......................X...X............X...............] duration: 11.532519284s trace: https://jaeger.dev.storj.io/trace/5527bfbc1f1a5fa4 FAIL (2025-12-10 14:32:33 UTC) orbiter warning: context deadline exceeded Duration: 11.532549333s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....X.......................X...X............X..............] duration: 12.638617922s trace: https://jaeger.dev.storj.io/trace/5527bfbc1f1a5fa4 FAIL (2025-12-10 14:32:11 UTC) orbiter warning: context deadline exceeded Duration: 12.63875383s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X.XX...XX.X...X...X...X..........X..X...X..X...X...X.X...] duration: 29.988015975s trace: https://jaeger.dev.storj.io/trace/7d0d581bad1b2bc6 FAIL (2025-12-10 14:32:08 UTC) orbiter warning: context deadline exceeded Duration: 29.988054919s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX.XXXXX.X] duration: 5.001335012s trace: https://jaeger.dev.storj.io/trace/2b997c4b34cac8c7 FAIL (2025-12-10 14:31:44 UTC) orbiter warning: context deadline exceeded Duration: 5.001358907s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....X.......................X...X............X.............] duration: 11.437448558s trace: https://jaeger.dev.storj.io/trace/5527bfbc1f1a5fa4 FAIL (2025-12-10 14:31:22 UTC) orbiter warning: context deadline exceeded Duration: 11.437483825s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX.XXXXX.] duration: 4.897401718s trace: https://jaeger.dev.storj.io/trace/3bea971ab6fdd911 FAIL (2025-12-10 14:31:08 UTC) orbiter warning: context deadline exceeded Duration: 4.89743409s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X.XX...XX.X...X...X...X..........X..X...X..X...X...X.X..] duration: 26.019053335s trace: https://jaeger.dev.storj.io/trace/7d0d581bad1b2bc6 FAIL (2025-12-10 14:30:53 UTC) orbiter warning: context deadline exceeded Duration: 26.019068143s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......X.......................X...X............X............] duration: 13.578234455s trace: https://jaeger.dev.storj.io/trace/5527bfbc1f1a5fa4 FAIL (2025-12-10 14:30:16 UTC) orbiter warning: context deadline exceeded Duration: 13.578267698s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX.XXXXX] duration: 5.001550517s trace: https://jaeger.dev.storj.io/trace/3bea971ab6fdd911 FAIL (2025-12-10 14:30:11 UTC) orbiter warning: context deadline exceeded Duration: 5.001578279s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X....X.XX...XX.X...X...X...X..........X..X...X..X...X...X.X.] duration: 22.659715861s trace: https://jaeger.dev.storj.io/trace/7d0d581bad1b2bc6 FAIL (2025-12-10 14:29:50 UTC) orbiter warning: context deadline exceeded Duration: 22.65973112s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 12.643711157s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:29:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 12.643777273s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X.......................X...X............X...........] duration: 14.021594001s trace: https://jaeger.dev.storj.io/trace/5527bfbc1f1a5fa4 FAIL (2025-12-10 14:29:34 UTC) orbiter warning: context deadline exceeded Duration: 14.021619098s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX....X.XX...XX.X...X...X...X..........X..X...X..X...X...X.X] duration: 30.001342958s trace: https://jaeger.dev.storj.io/trace/7d0d581bad1b2bc6 FAIL (2025-12-10 14:29:27 UTC) orbiter warning: context deadline exceeded Duration: 30.001366782s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX.XXXX] duration: 5.001631139s trace: https://jaeger.dev.storj.io/trace/255e547d6dc08edf FAIL (2025-12-10 14:29:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001658961s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 16.718208714s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:28:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 16.718237468s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........X.......................X...X............X..........] duration: 23.238410406s trace: https://jaeger.dev.storj.io/trace/5527bfbc1f1a5fa4 FAIL (2025-12-10 14:28:53 UTC) orbiter warning: context deadline exceeded Duration: 23.238436494s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXX....X.XX...XX.X...X...X...X..........X..X...X..X...X...X.] duration: 24.421067651s trace: https://jaeger.dev.storj.io/trace/485157a6a8dc8d02 FAIL (2025-12-10 14:28:41 UTC) orbiter warning: context deadline exceeded Duration: 24.421094411s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX.XXX] duration: 5.001652821s trace: https://jaeger.dev.storj.io/trace/48b5277af1ba1a7 FAIL (2025-12-10 14:28:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00167936s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........X.......................X...X............X.........] duration: 15.945993928s trace: https://jaeger.dev.storj.io/trace/5527bfbc1f1a5fa4 FAIL (2025-12-10 14:28:27 UTC) orbiter warning: context deadline exceeded Duration: 15.946016971s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........X.......................X...X............X........] duration: 20.868341485s trace: https://jaeger.dev.storj.io/trace/5527bfbc1f1a5fa4 FAIL (2025-12-10 14:28:08 UTC) orbiter warning: context deadline exceeded Duration: 20.86836955s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX.XX] duration: 5.001184075s trace: https://jaeger.dev.storj.io/trace/3ac155e6c1873934 FAIL (2025-12-10 14:27:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001204754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 18.884608693s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:27:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 18.884637749s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXXX....X.XX...XX.X...X...X...X..........X..X...X..X...X...X] duration: 30.001048243s trace: https://jaeger.dev.storj.io/trace/485157a6a8dc8d02 FAIL (2025-12-10 14:27:40 UTC) orbiter warning: context deadline exceeded Duration: 30.001068551s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XXXX....X.XX...XX.X...X...X...X..........X..X...X..X...X...] duration: 29.619916988s trace: https://jaeger.dev.storj.io/trace/1068d8d9f57bbf5f FAIL (2025-12-10 14:27:01 UTC) orbiter warning: context deadline exceeded Duration: 29.619935093s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........X.......................X...X............X.......] duration: 17.726886583s trace: https://jaeger.dev.storj.io/trace/5527bfbc1f1a5fa4 FAIL (2025-12-10 14:26:58 UTC) orbiter warning: context deadline exceeded Duration: 17.72693629s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX.X] duration: 5.001036058s trace: https://jaeger.dev.storj.io/trace/3228ce549b5a4489 FAIL (2025-12-10 14:26:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001058991s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.XXXX....X.XX...XX.X...X...X...X..........X..X...X..X...X..] duration: 25.25217431s trace: https://jaeger.dev.storj.io/trace/1068d8d9f57bbf5f FAIL (2025-12-10 14:26:31 UTC) orbiter warning: context deadline exceeded Duration: 25.252204376s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 16.394767511s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:26:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 16.394796626s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............X.......................X...X............X......] duration: 14.926635369s trace: https://jaeger.dev.storj.io/trace/5527bfbc1f1a5fa4 FAIL (2025-12-10 14:26:20 UTC) orbiter warning: context deadline exceeded Duration: 14.926685104s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX.] duration: 4.157848038s trace: https://jaeger.dev.storj.io/trace/15ca3a8801d40d44 FAIL (2025-12-10 14:25:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.157876131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.XXXX....X.XX...XX.X...X...X...X..........X..X...X..X...X.] duration: 22.259354978s trace: https://jaeger.dev.storj.io/trace/1068d8d9f57bbf5f FAIL (2025-12-10 14:25:36 UTC) orbiter warning: context deadline exceeded Duration: 22.259410253s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 20.726260599s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:25:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 20.726289403s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............X.......................X...X............X.....] duration: 18.11360374s trace: https://jaeger.dev.storj.io/trace/5527bfbc1f1a5fa4 FAIL (2025-12-10 14:25:18 UTC) orbiter warning: context deadline exceeded Duration: 18.113669466s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X.XXXX....X.XX...XX.X...X...X...X..........X..X...X..X...X] duration: 30.0018433s trace: https://jaeger.dev.storj.io/trace/1068d8d9f57bbf5f FAIL (2025-12-10 14:25:13 UTC) orbiter warning: context deadline exceeded Duration: 30.001867337s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX] duration: 5.002585808s trace: https://jaeger.dev.storj.io/trace/15ca3a8801d40d44 FAIL (2025-12-10 14:24:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002611678s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 17.531783107s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:24:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 17.531811651s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXX] duration: 5.001503348s trace: https://jaeger.dev.storj.io/trace/32334db0de4d451 FAIL (2025-12-10 14:24:40 UTC) orbiter warning: context deadline exceeded Duration: 5.00152552s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X.XXXX....X.XX...XX.X...X...X...X..........X..X...X..X...] duration: 22.717075876s trace: https://jaeger.dev.storj.io/trace/3af62d7900bdd39a FAIL (2025-12-10 14:24:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.717103038s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............X.......................X...X............X....] duration: 13.116362654s trace: https://jaeger.dev.storj.io/trace/5527bfbc1f1a5fa4 FAIL (2025-12-10 14:24:18 UTC) orbiter warning: context deadline exceeded Duration: 13.116392781s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...............X.......................X...X............X...] duration: 13.073457662s trace: https://jaeger.dev.storj.io/trace/5527bfbc1f1a5fa4 FAIL (2025-12-10 14:23:51 UTC) orbiter warning: context deadline exceeded Duration: 13.073484524s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....X.XXXX....X.XX...XX.X...X...X...X..........X..X...X..X..] duration: 28.467514523s trace: https://jaeger.dev.storj.io/trace/3af62d7900bdd39a FAIL (2025-12-10 14:23:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.467542626s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 10.608241955s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:23:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 10.608269307s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXX] duration: 5.001328308s trace: https://jaeger.dev.storj.io/trace/e4e2ad9bd8dfcb3 FAIL (2025-12-10 14:23:33 UTC) orbiter warning: context deadline exceeded Duration: 5.00135025s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X....X.XXXX....X.XX...XX.X...X...X...X..........X..X...X..X.] duration: 18.211205624s trace: https://jaeger.dev.storj.io/trace/3af62d7900bdd39a FAIL (2025-12-10 14:23:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.211233438s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [................X.......................X...X............X..] duration: 19.307001094s trace: https://jaeger.dev.storj.io/trace/5527bfbc1f1a5fa4 FAIL (2025-12-10 14:22:44 UTC) orbiter warning: context deadline exceeded Duration: 19.307013728s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 8.359290979s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:22:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 8.359324622s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.................X.......................X...X............X.] duration: 42.435881948s trace: https://jaeger.dev.storj.io/trace/5527bfbc1f1a5fa4 FAIL (2025-12-10 14:22:25 UTC) orbiter warning: context deadline exceeded Duration: 42.435900723s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X....X.XXXX....X.XX...XX.X...X...X...X..........X..X...X..X] duration: 30.002315188s trace: https://jaeger.dev.storj.io/trace/3af62d7900bdd39a FAIL (2025-12-10 14:22:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002349895s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXX] duration: 5.001495618s trace: https://jaeger.dev.storj.io/trace/3bfce95e1899e636 FAIL (2025-12-10 14:22:20 UTC) orbiter warning: context deadline exceeded Duration: 5.001519795s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 10.854693609s trace: https://jaeger.dev.storj.io/trace/3b8d3b772c1b7e70 FAIL (2025-12-10 14:21:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ) context deadline exceeded ======LOGS====== Dec 10 13:39:27.980451296 (118h10m27.851534331s) [upload=103151]: Starting upload Dec 10 13:39:27.980533502 (118h10m27.851616547s) [upload=103151]: Done starting upload Dec 10 13:39:27.980702654 (118h10m27.851785699s) [upload=103151]: Uploading segments... Dec 10 13:39:27.980851888 (118h10m27.851934933s) [upload=103151]: Got next segment. Inline: false Dec 10 13:39:27.980915510 (118h10m27.851998555s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:39:27.980923284 (118h10m27.852006319s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:39:28.328765369 (118h10m28.199848414s) [seg_pos={0 0}, upload=103151]: Begin upload segment... Dec 10 13:39:28.328820373 (118h10m28.199903408s) [seg_pos={0 0}, upload=103151]: Done begin upload segment. Dec 10 13:39:28.328856151 (118h10m28.199939186s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.328867674 (118h10m28.199950709s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329607472 (118h10m28.200690528s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.329754152 (118h10m28.200837207s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330197757 (118h10m28.201280802s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.330761259 (118h10m28.201844304s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331110735 (118h10m28.20219378s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331464749 (118h10m28.202547784s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.331783286 (118h10m28.202866321s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332101542 (118h10m28.203184577s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.332434065 (118h10m28.2035171s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.333431755 (118h10m28.20451479s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.334509297 (118h10m28.205592342s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335134908 (118h10m28.206217943s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.335674264 (118h10m28.206757299s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:28.336746276 (118h10m28.207829321s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Uploading piece... Dec 10 13:39:29.488033344 (118h10m29.359116389s) [noise=true , address=77.90.31.19:4841 , piece_id=NCDZJBAW6RH54AGMRAUPZAQ5E6EDHCHZJH3VHKVCTDR3YGHXOM4A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 10 13:39:30.285902114 (118h10m30.156985159s) [upload=103151]: Next returned nil segment Dec 10 13:39:30.285938784 (118h10m30.157021819s) [upload=103151]: Waiting for error group managing segments... Dec 10 13:39:30.597009717 (118h10m30.468092763s) [noise=true , address=77.90.31.10:4520 , piece_id=GOZIIJPYNUE2JOBXEKXIZND63V2CPS6R3MMAFQD25FTDB7XRBN7Q, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:30.735873502 (118h10m30.606956547s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQI5HZTL5YFMAR6WNYIGFO24QIDPGWCNE5S3POHKETPL5OCANYPQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.319337539 (118h10m31.190420594s) [noise=true , address=145.79.227.40:4050 , piece_id=EPNWE2F6UE3XT4XGNLLKJTYZGSGZR6FN77HL7J3RXC2ZJRVKE6YQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.330611717 (118h10m31.201694752s) [noise=true , address=145.79.227.31:4797 , piece_id=WNSJD7FJFRVLVW4D5UPJH5WVLCHCNFYDFL3JD7VXUDARUH3QBGRA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.346897319 (118h10m31.217980354s) [noise=true , address=145.79.227.24:4583 , piece_id=C2TWWS6TZJENMRJRO3G63VNAFROSFYUNDXPM2YBRD2EV7QZLW6LA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:31.371629855 (118h10m31.2427129s) [noise=true , address=145.79.227.22:4510 , piece_id=DULM67MLGUIRNKF26UYNZ4KGZGJ4OO3JDM5O7ACWL5Z5QGFJ7WZA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.040038744 (118h10m31.911121789s) [noise=true , address=145.79.227.18:4006 , piece_id=PSIPYYQKKXJRRDN76DZWWKZVXBDPBHL7SFAX5KZ6WUJZS4AC5TWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.047018676 (118h10m31.918101721s) [noise=true , address=145.79.227.38:4977 , piece_id=SD2GG5N3HERZTFPORLSUQHWDYAYBAJMMHRNMX27WU3HAOMCAHCOQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.051258549 (118h10m31.922341614s) [noise=true , address=145.79.227.28:4727 , piece_id=3U7WLWI2WB35TXD2ZACV4H3E6EZ67BJP5P3IIT7E6SXFFEKUS7BQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.340671892 (118h10m32.211754947s) [noise=true , address=145.79.227.23:4546 , piece_id=BLFQLSLWMNSD3IPODHVSE3VQ3BE5SJZUBT2LXNJA2U7MEAQ3V4PQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:32.342510924 (118h10m32.213593959s) [noise=true , address=145.79.227.26:4655 , piece_id=25JJUTTTY253VRFRL6JO5DLF75FHXOPBTQGBNFB7XTHX5JRSECYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.162014519 (118h10m33.033097564s) [noise=true , address=145.79.227.32:4836 , piece_id=R4ZFAYZZVF6MQHLAIHSSZVKVD4FZNCDX75GFUZ63K6HMC4LRRLSQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464127443 (118h10m33.335210478s) [noise=true , address=77.90.31.12:4590 , piece_id=FUJOFQ35J2A3QH3H4C26GUD2FFCEBF35CSAIYR5B4XIKEW7Y4RWA, seg_pos={0 0}, upload=103151]: Done uploading piece. err: <nil> Dec 10 13:39:33.464170696 (118h10m33.335253731s) [seg_pos={0 0}, upload=103151]: Segment reached optimal threshold of 13 pieces. Dec 10 13:39:33.464522617 (118h10m33.335605652s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35882->145.79.227.14:4839: use of closed network connection Dec 10 13:39:33.464537104 (118h10m33.335620139s) [noise=true , address=145.79.227.14:4839 , piece_id=RQL3FSLEWDEL56DCGZGU3NCXKPKOPRIDXQSPACZR44Z7Z26ELYYQ, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464773714 (118h10m33.33585676s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 10 13:39:33.464785717 (118h10m33.335868762s) [noise=true , address=145.79.227.10:4401 , piece_id=GCLCANHVAXDKTJZ655Z34DXVLZEI7LCNHJEDOKTJX5QSXEHAUK6A, seg_pos={0 0}, upload=103151]: Long tail cancellation due to optimal threshold Dec 10 13:39:33.464817688 (118h10m33.335900733s) [seg_pos={0 0}, upload=103151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:39:33.464920033 (118h10m33.336003068s) [upload=103151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:39:33.464926966 (118h10m33.336010011s) [upload=103151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:39:33.902471702 (118h10m33.773554747s) [upload=103151]: Done uploading segments... Dec 10 13:39:33.902491529 (118h10m33.773574565s) [upload=103151]: Upload finished. err: <nil> Duration: 10.854724256s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)