Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [X............X..............................................] duration:
17.981900817s trace:
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:58:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 17.981915214s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
2.305146823s trace:
https://jaeger.dev.storj.io/trace/59c3b821b75eba3a
FAIL (2025-12-31 07:58:16 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 2.305174326s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X............X.............................................] duration:
49.584152612s trace:
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:58:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 49.584180656s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [............................X.........X............X..X.....] duration:
3.447082412s trace:
https://jaeger.dev.storj.io/trace/e2308925a42e319
FAIL (2025-12-31 07:58:12 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.447132497s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X.X.X.X......XXX.X....XXX..X...........X......X.X.X.XX....X.] duration:
6.58601134s trace:
https://jaeger.dev.storj.io/trace/4fc89fb7e2e6dbda
FAIL (2025-12-31 07:58:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 07:57:08.447503483 (616h28m8.318586528s) [upload=540239]: Starting upload
Dec 31 07:57:08.447610507 (616h28m8.318693542s) [upload=540239]: Done starting upload
Dec 31 07:57:08.447948941 (616h28m8.319031976s) [upload=540239]: Uploading segments...
Dec 31 07:57:08.448029134 (616h28m8.319112189s) [upload=540239]: Got next segment. Inline: false
Dec 31 07:57:08.448072366 (616h28m8.319155411s) [upload=540239]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:57:08.448077877 (616h28m8.319160912s) [upload=540239]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:57:08.694569526 (616h28m8.565652561s) [seg_pos={0 0}, upload=540239]: Begin upload segment...
Dec 31 07:57:08.694662322 (616h28m8.565745367s) [seg_pos={0 0}, upload=540239]: Done begin upload segment.
Dec 31 07:57:08.694724791 (616h28m8.565807826s) [noise=true , address=145.79.227.14:4839 , piece_id=ZLRU5YO5AT2PRTGELXAYSWXSWTEAGIGF5D2QVYW32TBEU64LSEAA, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.694797249 (616h28m8.565880284s) [noise=true , address=145.79.227.25:4618 , piece_id=T6677P6B7D4AYTXY2V62YWX25IYRBTMNRBQZNPU3VM3PK5MQFJTQ, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.695652859 (616h28m8.566735904s) [noise=true , address=77.90.31.21:4880 , piece_id=UDY2IXVJNMBELBRVARTRPZSPC4M4P4MQ36PRUBWS7HXRZLPSIAJA, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.695909297 (616h28m8.566992332s) [noise=true , address=145.79.227.36:4519 , piece_id=LFTJPQ6A2D2WFNPJZF75W5SUOU2Z2SVNGCD3KWWXVSYLW2R2B5ZQ, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.695978649 (616h28m8.567061684s) [noise=true , address=45.91.116.5:4297 , piece_id=IE43P5QJU2POO7WFESP2RLE33O3ZJ3ZXIF5QQDN3RR5ZZXQ7UUFA, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.696287176 (616h28m8.567370211s) [noise=true , address=145.79.227.51:4411 , piece_id=JYFXNOYLC7SONLD3PBHRTHCXPQRGVGMOA3GSVTJRSTMFZFNTKYNQ, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.696439155 (616h28m8.56752219s) [noise=true , address=145.79.227.46:4602 , piece_id=F5AS2TAIYU7EP44ZK5YT4XQZST6IZWKXNCZK5PFLQEFZ4QHBMIRQ, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.696544696 (616h28m8.567627731s) [noise=true , address=145.79.227.11:4731 , piece_id=5WJOOTXPQCCZ6RFBJFNLJP7J2JC3JGTALW3KRRQ3AIBBPMD7QARA, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.696839107 (616h28m8.567922143s) [noise=true , address=145.79.227.52:4445 , piece_id=6LFJBXARHGJ7LP4HXZ7WR6PUO3ZV5YLJ4X4PJOKHX2KFZKKXBMYQ, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.696979635 (616h28m8.56806268s) [noise=true , address=145.79.227.38:4978 , piece_id=QKBHRTHDFFBSKVYX2OWKLNQTSOVAOZM3D72574JRTRTJV6RXUUSQ, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.697138166 (616h28m8.568221201s) [noise=true , address=77.90.31.15:4697 , piece_id=LP2KN6JKVAUXHKXK757CMHQUZZKMUTI5V3G2R7IVVMA4HVG3VMJQ, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.697623430 (616h28m8.568706475s) [noise=true , address=145.79.227.31:4799 , piece_id=FFEZ33R54JTKGUYEZU35UVV2OH2IK4WLXWZMFLULWDCIRFIJSB4A, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.698177425 (616h28m8.56926046s) [noise=true , address=77.90.31.11:4554 , piece_id=B2XFUFJHU32ZJNO7TZAQ3GUI2SGCY7CUXTT5LY4U4CN4IN3R4IQA, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.698805591 (616h28m8.569888627s) [noise=true , address=77.90.31.10:4519 , piece_id=35OZIXPVY44CBSIT6J2OPL6DCMA4E3WNTKAGDQUQMT4RT6VH3QPQ, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.702485840 (616h28m8.573568875s) [noise=true , address=77.90.31.19:4844 , piece_id=VXIVMHONESWRCJ76EBJMSSETKQDXLJTLPI7CP6L5X3URF36OHRDQ, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.703118644 (616h28m8.574201679s) [noise=true , address=145.79.227.34:4906 , piece_id=2BYTLI7TEKLDBGQYZMB4YW2WDRUUEPPNAMAHMSHBCOAU6RACN5AQ, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:10.401066406 (616h28m10.272149441s) [upload=540239]: Next returned nil segment
Dec 31 07:57:10.401081094 (616h28m10.272164129s) [upload=540239]: Waiting for error group managing segments...
Dec 31 07:57:10.688568696 (616h28m10.559651741s) [noise=true , address=145.79.227.38:4978 , piece_id=QKBHRTHDFFBSKVYX2OWKLNQTSOVAOZM3D72574JRTRTJV6RXUUSQ, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:10.688596779 (616h28m10.559679814s) [noise=true , address=145.79.227.31:4799 , piece_id=FFEZ33R54JTKGUYEZU35UVV2OH2IK4WLXWZMFLULWDCIRFIJSB4A, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:10.690066427 (616h28m10.561149462s) [noise=true , address=145.79.227.11:4731 , piece_id=5WJOOTXPQCCZ6RFBJFNLJP7J2JC3JGTALW3KRRQ3AIBBPMD7QARA, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:10.690429960 (616h28m10.561512995s) [noise=true , address=145.79.227.46:4602 , piece_id=F5AS2TAIYU7EP44ZK5YT4XQZST6IZWKXNCZK5PFLQEFZ4QHBMIRQ, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:10.698852639 (616h28m10.569935674s) [noise=true , address=145.79.227.36:4519 , piece_id=LFTJPQ6A2D2WFNPJZF75W5SUOU2Z2SVNGCD3KWWXVSYLW2R2B5ZQ, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:10.700381039 (616h28m10.571464084s) [noise=true , address=145.79.227.52:4445 , piece_id=6LFJBXARHGJ7LP4HXZ7WR6PUO3ZV5YLJ4X4PJOKHX2KFZKKXBMYQ, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:10.704769646 (616h28m10.575852681s) [noise=true , address=145.79.227.14:4839 , piece_id=ZLRU5YO5AT2PRTGELXAYSWXSWTEAGIGF5D2QVYW32TBEU64LSEAA, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:10.706647371 (616h28m10.577730406s) [noise=true , address=145.79.227.25:4618 , piece_id=T6677P6B7D4AYTXY2V62YWX25IYRBTMNRBQZNPU3VM3PK5MQFJTQ, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:10.765713922 (616h28m10.636796977s) [noise=true , address=145.79.227.51:4411 , piece_id=JYFXNOYLC7SONLD3PBHRTHCXPQRGVGMOA3GSVTJRSTMFZFNTKYNQ, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:10.856821355 (616h28m10.7279044s) [noise=true , address=77.90.31.11:4554 , piece_id=B2XFUFJHU32ZJNO7TZAQ3GUI2SGCY7CUXTT5LY4U4CN4IN3R4IQA, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:11.026174881 (616h28m10.897257946s) [noise=true , address=77.90.31.15:4697 , piece_id=LP2KN6JKVAUXHKXK757CMHQUZZKMUTI5V3G2R7IVVMA4HVG3VMJQ, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:11.184092080 (616h28m11.055175125s) [noise=true , address=77.90.31.19:4844 , piece_id=VXIVMHONESWRCJ76EBJMSSETKQDXLJTLPI7CP6L5X3URF36OHRDQ, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:11.685168189 (616h28m11.556251224s) [noise=true , address=77.90.31.21:4880 , piece_id=UDY2IXVJNMBELBRVARTRPZSPC4M4P4MQ36PRUBWS7HXRZLPSIAJA, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:11.685193847 (616h28m11.556276882s) [seg_pos={0 0}, upload=540239]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:57:11.685405590 (616h28m11.556488625s) [noise=true , address=77.90.31.10:4519 , piece_id=35OZIXPVY44CBSIT6J2OPL6DCMA4E3WNTKAGDQUQMT4RT6VH3QPQ, seg_pos={0 0}, upload=540239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:48200->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:48200->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48200->77.90.31.10:4519: use of closed network connection
Dec 31 07:57:11.685414778 (616h28m11.556497823s) [noise=true , address=77.90.31.10:4519 , piece_id=35OZIXPVY44CBSIT6J2OPL6DCMA4E3WNTKAGDQUQMT4RT6VH3QPQ, seg_pos={0 0}, upload=540239]: Long tail cancellation due to optimal threshold
Dec 31 07:57:11.685477086 (616h28m11.556560121s) [noise=true , address=45.91.116.5:4297 , piece_id=IE43P5QJU2POO7WFESP2RLE33O3ZJ3ZXIF5QQDN3RR5ZZXQ7UUFA, seg_pos={0 0}, upload=540239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Dec 31 07:57:11.685493057 (616h28m11.556576092s) [noise=true , address=45.91.116.5:4297 , piece_id=IE43P5QJU2POO7WFESP2RLE33O3ZJ3ZXIF5QQDN3RR5ZZXQ7UUFA, seg_pos={0 0}, upload=540239]: Long tail cancellation due to optimal threshold
Dec 31 07:57:11.685831181 (616h28m11.556914226s) [noise=true , address=145.79.227.34:4906 , piece_id=2BYTLI7TEKLDBGQYZMB4YW2WDRUUEPPNAMAHMSHBCOAU6RACN5AQ, seg_pos={0 0}, upload=540239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 31 07:57:11.685838455 (616h28m11.5569215s) [noise=true , address=145.79.227.34:4906 , piece_id=2BYTLI7TEKLDBGQYZMB4YW2WDRUUEPPNAMAHMSHBCOAU6RACN5AQ, seg_pos={0 0}, upload=540239]: Long tail cancellation due to optimal threshold
Dec 31 07:57:11.685859134 (616h28m11.556942169s) [seg_pos={0 0}, upload=540239]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:57:11.685955498 (616h28m11.557038533s) [upload=540239]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:57:11.685961439 (616h28m11.557044474s) [upload=540239]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:57:11.977681059 (616h28m11.848764114s) [upload=540239]: Done uploading segments...
Dec 31 07:57:11.977706598 (616h28m11.848789663s) [upload=540239]: Upload finished. err: <nil>
Duration: 6.586055715s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
12.934135311s trace:
https://jaeger.dev.storj.io/trace/5dd0609e3a2ac547
FAIL (2025-12-31 07:57:59 UTC)
orbiter warning: context deadline exceeded
Duration: 12.934164576s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
10.259406121s trace:
https://jaeger.dev.storj.io/trace/5dd0609e3a2ac547
FAIL (2025-12-31 07:57:40 UTC)
orbiter warning: context deadline exceeded
Duration: 10.25942681s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.X.X.X.X......XXX.X....XXX..X...........X......X.X.X.XX....X] duration:
30.003974873s trace:
https://jaeger.dev.storj.io/trace/4fc89fb7e2e6dbda
FAIL (2025-12-31 07:57:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 07:57:08.447503483 (616h28m8.318586528s) [upload=540239]: Starting upload
Dec 31 07:57:08.447610507 (616h28m8.318693542s) [upload=540239]: Done starting upload
Dec 31 07:57:08.447948941 (616h28m8.319031976s) [upload=540239]: Uploading segments...
Dec 31 07:57:08.448029134 (616h28m8.319112189s) [upload=540239]: Got next segment. Inline: false
Dec 31 07:57:08.448072366 (616h28m8.319155411s) [upload=540239]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:57:08.448077877 (616h28m8.319160912s) [upload=540239]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:57:08.694569526 (616h28m8.565652561s) [seg_pos={0 0}, upload=540239]: Begin upload segment...
Dec 31 07:57:08.694662322 (616h28m8.565745367s) [seg_pos={0 0}, upload=540239]: Done begin upload segment.
Dec 31 07:57:08.694724791 (616h28m8.565807826s) [noise=true , address=145.79.227.14:4839 , piece_id=ZLRU5YO5AT2PRTGELXAYSWXSWTEAGIGF5D2QVYW32TBEU64LSEAA, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.694797249 (616h28m8.565880284s) [noise=true , address=145.79.227.25:4618 , piece_id=T6677P6B7D4AYTXY2V62YWX25IYRBTMNRBQZNPU3VM3PK5MQFJTQ, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.695652859 (616h28m8.566735904s) [noise=true , address=77.90.31.21:4880 , piece_id=UDY2IXVJNMBELBRVARTRPZSPC4M4P4MQ36PRUBWS7HXRZLPSIAJA, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.695909297 (616h28m8.566992332s) [noise=true , address=145.79.227.36:4519 , piece_id=LFTJPQ6A2D2WFNPJZF75W5SUOU2Z2SVNGCD3KWWXVSYLW2R2B5ZQ, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.695978649 (616h28m8.567061684s) [noise=true , address=45.91.116.5:4297 , piece_id=IE43P5QJU2POO7WFESP2RLE33O3ZJ3ZXIF5QQDN3RR5ZZXQ7UUFA, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.696287176 (616h28m8.567370211s) [noise=true , address=145.79.227.51:4411 , piece_id=JYFXNOYLC7SONLD3PBHRTHCXPQRGVGMOA3GSVTJRSTMFZFNTKYNQ, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.696439155 (616h28m8.56752219s) [noise=true , address=145.79.227.46:4602 , piece_id=F5AS2TAIYU7EP44ZK5YT4XQZST6IZWKXNCZK5PFLQEFZ4QHBMIRQ, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.696544696 (616h28m8.567627731s) [noise=true , address=145.79.227.11:4731 , piece_id=5WJOOTXPQCCZ6RFBJFNLJP7J2JC3JGTALW3KRRQ3AIBBPMD7QARA, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.696839107 (616h28m8.567922143s) [noise=true , address=145.79.227.52:4445 , piece_id=6LFJBXARHGJ7LP4HXZ7WR6PUO3ZV5YLJ4X4PJOKHX2KFZKKXBMYQ, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.696979635 (616h28m8.56806268s) [noise=true , address=145.79.227.38:4978 , piece_id=QKBHRTHDFFBSKVYX2OWKLNQTSOVAOZM3D72574JRTRTJV6RXUUSQ, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.697138166 (616h28m8.568221201s) [noise=true , address=77.90.31.15:4697 , piece_id=LP2KN6JKVAUXHKXK757CMHQUZZKMUTI5V3G2R7IVVMA4HVG3VMJQ, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.697623430 (616h28m8.568706475s) [noise=true , address=145.79.227.31:4799 , piece_id=FFEZ33R54JTKGUYEZU35UVV2OH2IK4WLXWZMFLULWDCIRFIJSB4A, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.698177425 (616h28m8.56926046s) [noise=true , address=77.90.31.11:4554 , piece_id=B2XFUFJHU32ZJNO7TZAQ3GUI2SGCY7CUXTT5LY4U4CN4IN3R4IQA, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.698805591 (616h28m8.569888627s) [noise=true , address=77.90.31.10:4519 , piece_id=35OZIXPVY44CBSIT6J2OPL6DCMA4E3WNTKAGDQUQMT4RT6VH3QPQ, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.702485840 (616h28m8.573568875s) [noise=true , address=77.90.31.19:4844 , piece_id=VXIVMHONESWRCJ76EBJMSSETKQDXLJTLPI7CP6L5X3URF36OHRDQ, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:08.703118644 (616h28m8.574201679s) [noise=true , address=145.79.227.34:4906 , piece_id=2BYTLI7TEKLDBGQYZMB4YW2WDRUUEPPNAMAHMSHBCOAU6RACN5AQ, seg_pos={0 0}, upload=540239]: Uploading piece...
Dec 31 07:57:10.401066406 (616h28m10.272149441s) [upload=540239]: Next returned nil segment
Dec 31 07:57:10.401081094 (616h28m10.272164129s) [upload=540239]: Waiting for error group managing segments...
Dec 31 07:57:10.688568696 (616h28m10.559651741s) [noise=true , address=145.79.227.38:4978 , piece_id=QKBHRTHDFFBSKVYX2OWKLNQTSOVAOZM3D72574JRTRTJV6RXUUSQ, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:10.688596779 (616h28m10.559679814s) [noise=true , address=145.79.227.31:4799 , piece_id=FFEZ33R54JTKGUYEZU35UVV2OH2IK4WLXWZMFLULWDCIRFIJSB4A, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:10.690066427 (616h28m10.561149462s) [noise=true , address=145.79.227.11:4731 , piece_id=5WJOOTXPQCCZ6RFBJFNLJP7J2JC3JGTALW3KRRQ3AIBBPMD7QARA, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:10.690429960 (616h28m10.561512995s) [noise=true , address=145.79.227.46:4602 , piece_id=F5AS2TAIYU7EP44ZK5YT4XQZST6IZWKXNCZK5PFLQEFZ4QHBMIRQ, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:10.698852639 (616h28m10.569935674s) [noise=true , address=145.79.227.36:4519 , piece_id=LFTJPQ6A2D2WFNPJZF75W5SUOU2Z2SVNGCD3KWWXVSYLW2R2B5ZQ, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:10.700381039 (616h28m10.571464084s) [noise=true , address=145.79.227.52:4445 , piece_id=6LFJBXARHGJ7LP4HXZ7WR6PUO3ZV5YLJ4X4PJOKHX2KFZKKXBMYQ, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:10.704769646 (616h28m10.575852681s) [noise=true , address=145.79.227.14:4839 , piece_id=ZLRU5YO5AT2PRTGELXAYSWXSWTEAGIGF5D2QVYW32TBEU64LSEAA, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:10.706647371 (616h28m10.577730406s) [noise=true , address=145.79.227.25:4618 , piece_id=T6677P6B7D4AYTXY2V62YWX25IYRBTMNRBQZNPU3VM3PK5MQFJTQ, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:10.765713922 (616h28m10.636796977s) [noise=true , address=145.79.227.51:4411 , piece_id=JYFXNOYLC7SONLD3PBHRTHCXPQRGVGMOA3GSVTJRSTMFZFNTKYNQ, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:10.856821355 (616h28m10.7279044s) [noise=true , address=77.90.31.11:4554 , piece_id=B2XFUFJHU32ZJNO7TZAQ3GUI2SGCY7CUXTT5LY4U4CN4IN3R4IQA, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:11.026174881 (616h28m10.897257946s) [noise=true , address=77.90.31.15:4697 , piece_id=LP2KN6JKVAUXHKXK757CMHQUZZKMUTI5V3G2R7IVVMA4HVG3VMJQ, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:11.184092080 (616h28m11.055175125s) [noise=true , address=77.90.31.19:4844 , piece_id=VXIVMHONESWRCJ76EBJMSSETKQDXLJTLPI7CP6L5X3URF36OHRDQ, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:11.685168189 (616h28m11.556251224s) [noise=true , address=77.90.31.21:4880 , piece_id=UDY2IXVJNMBELBRVARTRPZSPC4M4P4MQ36PRUBWS7HXRZLPSIAJA, seg_pos={0 0}, upload=540239]: Done uploading piece. err: <nil>
Dec 31 07:57:11.685193847 (616h28m11.556276882s) [seg_pos={0 0}, upload=540239]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:57:11.685405590 (616h28m11.556488625s) [noise=true , address=77.90.31.10:4519 , piece_id=35OZIXPVY44CBSIT6J2OPL6DCMA4E3WNTKAGDQUQMT4RT6VH3QPQ, seg_pos={0 0}, upload=540239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:48200->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:48200->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48200->77.90.31.10:4519: use of closed network connection
Dec 31 07:57:11.685414778 (616h28m11.556497823s) [noise=true , address=77.90.31.10:4519 , piece_id=35OZIXPVY44CBSIT6J2OPL6DCMA4E3WNTKAGDQUQMT4RT6VH3QPQ, seg_pos={0 0}, upload=540239]: Long tail cancellation due to optimal threshold
Dec 31 07:57:11.685477086 (616h28m11.556560121s) [noise=true , address=45.91.116.5:4297 , piece_id=IE43P5QJU2POO7WFESP2RLE33O3ZJ3ZXIF5QQDN3RR5ZZXQ7UUFA, seg_pos={0 0}, upload=540239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Dec 31 07:57:11.685493057 (616h28m11.556576092s) [noise=true , address=45.91.116.5:4297 , piece_id=IE43P5QJU2POO7WFESP2RLE33O3ZJ3ZXIF5QQDN3RR5ZZXQ7UUFA, seg_pos={0 0}, upload=540239]: Long tail cancellation due to optimal threshold
Dec 31 07:57:11.685831181 (616h28m11.556914226s) [noise=true , address=145.79.227.34:4906 , piece_id=2BYTLI7TEKLDBGQYZMB4YW2WDRUUEPPNAMAHMSHBCOAU6RACN5AQ, seg_pos={0 0}, upload=540239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 31 07:57:11.685838455 (616h28m11.5569215s) [noise=true , address=145.79.227.34:4906 , piece_id=2BYTLI7TEKLDBGQYZMB4YW2WDRUUEPPNAMAHMSHBCOAU6RACN5AQ, seg_pos={0 0}, upload=540239]: Long tail cancellation due to optimal threshold
Dec 31 07:57:11.685859134 (616h28m11.556942169s) [seg_pos={0 0}, upload=540239]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:57:11.685955498 (616h28m11.557038533s) [upload=540239]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:57:11.685961439 (616h28m11.557044474s) [upload=540239]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:57:11.977681059 (616h28m11.848764114s) [upload=540239]: Done uploading segments...
Dec 31 07:57:11.977706598 (616h28m11.848789663s) [upload=540239]: Upload finished. err: <nil>
Duration: 30.003998096s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.............................X.........X............X..X....] duration:
5.000801927s trace:
https://jaeger.dev.storj.io/trace/e2308925a42e319
FAIL (2025-12-31 07:57:37 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000825682s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X............X............................................] duration:
30.575268791s trace:
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:57:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 30.575284031s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
4.348090183s trace:
https://jaeger.dev.storj.io/trace/59c3b821b75eba3a
FAIL (2025-12-31 07:57:05 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 4.34811969s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
6.186376297s trace:
https://jaeger.dev.storj.io/trace/5dd0609e3a2ac547
FAIL (2025-12-31 07:56:49 UTC)
orbiter warning: context deadline exceeded
Duration: 6.186402236s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X.X.X.X.X......XXX.X....XXX..X...........X......X.X.X.XX....] duration:
5.749941656s trace:
https://jaeger.dev.storj.io/trace/6c9f60d6c2a165da
FAIL (2025-12-31 07:56:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA) context deadline exceeded
======LOGS======
Dec 31 07:53:38.027484676 (616h24m37.898567721s) [upload=540182]: Starting upload
Dec 31 07:53:38.027534512 (616h24m37.898617547s) [upload=540182]: Done starting upload
Dec 31 07:53:38.027801700 (616h24m37.898884735s) [upload=540182]: Uploading segments...
Dec 31 07:53:38.027979689 (616h24m37.899062734s) [upload=540182]: Got next segment. Inline: false
Dec 31 07:53:38.028073868 (616h24m37.899156903s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:53:38.028082775 (616h24m37.89916581s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:53:38.277891170 (616h24m38.148974205s) [seg_pos={0 0}, upload=540182]: Begin upload segment...
Dec 31 07:53:38.277968046 (616h24m38.149051081s) [seg_pos={0 0}, upload=540182]: Done begin upload segment.
Dec 31 07:53:38.278029704 (616h24m38.149112749s) [noise=true , address=45.91.116.6:4975 , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278042758 (616h24m38.149125794s) [noise=true , address=77.90.31.12:4589 , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278144813 (616h24m38.149227848s) [noise=true , address=145.79.227.50:4375 , piece_id=CXKJCYVE6SJOOVQWHH34MCITFOFEV27SP4OKAPXF4DJT3EXIHWGQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278830739 (616h24m38.149913784s) [noise=true , address=77.90.31.19:4842 , piece_id=PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279143605 (616h24m38.15022664s) [noise=true , address=77.90.31.15:4700 , piece_id=ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279489213 (616h24m38.150572258s) [noise=true , address=45.91.116.13:4154 , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279878865 (616h24m38.15096191s) [noise=true , address=145.79.227.26:4655 , piece_id=YHNBMQCH6QZRCTHVFDZRVHMXB2BP3MJ7B3RG3QIPBQ4PQW5JYB6A, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.280269218 (616h24m38.151352253s) [noise=true , address=145.79.227.37:4566 , piece_id=6E73Y5FUFAGQLU4IYUQNJ2J7XXXRJQGPM7KJRHMYQ3ENEBN76QKQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.280653881 (616h24m38.151736926s) [noise=true , address=145.79.227.24:4581 , piece_id=IKQBNNGA2QEVTMZZMGPMHLLIKLEDKZCL34X2DDIFZFHW2GBKLCXA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.281852182 (616h24m38.152935227s) [noise=true , address=145.79.227.16:4910 , piece_id=6L6QC3KSIG6CVDPLG55O6Q24L3QZ72GL3OTPAETHJG7I4Z4WMYKA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.281889793 (616h24m38.152972828s) [noise=true , address=145.79.227.45:4229 , piece_id=3EKCB2QETOCEMR2WCFYDZKVYT2R3CBOOKMUHXBMANKMOJD7H7IZA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.283093776 (616h24m38.154176821s) [noise=true , address=145.79.227.12:4767 , piece_id=GGEX2BKWC4URKRI34IAR7Z2KAP62FVRDM3ELUTNIH6WB4QN3JLRQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.283497055 (616h24m38.15458009s) [noise=true , address=145.79.227.23:4545 , piece_id=F5KYOLWB2IZRRCKVYFGHFZ23UZV7FUR2PCF2HO7GSHEMKIY5IL3Q, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.284318098 (616h24m38.155401133s) [noise=true , address=145.79.227.51:4410 , piece_id=VYULD4SFKPQRYZSOPM2PUQC4LUQAQ4WXRYVSOSRYCA5N5LFV4FAA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.285603535 (616h24m38.15668657s) [noise=true , address=145.79.227.52:4446 , piece_id=6FI2W5DULY5FFDFIFXLNSDUX7EEVDGB3WLWIEEZXFKNOR3AIOSIQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.286960870 (616h24m38.158043915s) [noise=true , address=145.79.227.18:4006 , piece_id=LFDSYTOQR42MSDEIRDUXGRNBUF3ONMCL6BJTMRNLNDHHNY5V7ZDQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:39.976475363 (616h24m39.847558408s) [upload=540182]: Next returned nil segment
Dec 31 07:53:39.976503577 (616h24m39.847586622s) [upload=540182]: Waiting for error group managing segments...
Dec 31 07:53:40.263089662 (616h24m40.134172697s) [noise=true , address=145.79.227.52:4446 , piece_id=6FI2W5DULY5FFDFIFXLNSDUX7EEVDGB3WLWIEEZXFKNOR3AIOSIQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.263514351 (616h24m40.134597386s) [noise=true , address=145.79.227.16:4910 , piece_id=6L6QC3KSIG6CVDPLG55O6Q24L3QZ72GL3OTPAETHJG7I4Z4WMYKA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.266215304 (616h24m40.137298339s) [noise=true , address=145.79.227.12:4767 , piece_id=GGEX2BKWC4URKRI34IAR7Z2KAP62FVRDM3ELUTNIH6WB4QN3JLRQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.274366505 (616h24m40.14544955s) [noise=true , address=145.79.227.51:4410 , piece_id=VYULD4SFKPQRYZSOPM2PUQC4LUQAQ4WXRYVSOSRYCA5N5LFV4FAA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.277241870 (616h24m40.148324905s) [noise=true , address=145.79.227.26:4655 , piece_id=YHNBMQCH6QZRCTHVFDZRVHMXB2BP3MJ7B3RG3QIPBQ4PQW5JYB6A, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.277655779 (616h24m40.148738814s) [noise=true , address=145.79.227.37:4566 , piece_id=6E73Y5FUFAGQLU4IYUQNJ2J7XXXRJQGPM7KJRHMYQ3ENEBN76QKQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.288485420 (616h24m40.159568465s) [noise=true , address=145.79.227.50:4375 , piece_id=CXKJCYVE6SJOOVQWHH34MCITFOFEV27SP4OKAPXF4DJT3EXIHWGQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.290211387 (616h24m40.161294432s) [noise=true , address=145.79.227.18:4006 , piece_id=LFDSYTOQR42MSDEIRDUXGRNBUF3ONMCL6BJTMRNLNDHHNY5V7ZDQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.293396742 (616h24m40.164479787s) [noise=true , address=145.79.227.24:4581 , piece_id=IKQBNNGA2QEVTMZZMGPMHLLIKLEDKZCL34X2DDIFZFHW2GBKLCXA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.302471963 (616h24m40.173554998s) [noise=true , address=145.79.227.45:4229 , piece_id=3EKCB2QETOCEMR2WCFYDZKVYT2R3CBOOKMUHXBMANKMOJD7H7IZA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.354700213 (616h24m40.225783248s) [noise=true , address=145.79.227.23:4545 , piece_id=F5KYOLWB2IZRRCKVYFGHFZ23UZV7FUR2PCF2HO7GSHEMKIY5IL3Q, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.453140008 (616h24m40.324223043s) [noise=true , address=77.90.31.19:4842 , piece_id=PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.527310551 (616h24m40.398393596s) [noise=true , address=77.90.31.15:4700 , piece_id=ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.527327724 (616h24m40.398410759s) [seg_pos={0 0}, upload=540182]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:53:40.527495442 (616h24m40.398578477s) [noise=true , address=45.91.116.13:4154 , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 31 07:53:40.527501184 (616h24m40.398584219s) [noise=true , address=45.91.116.13:4154 , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.527703939 (616h24m40.398786974s) [noise=true , address=77.90.31.12:4589 , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection
Dec 31 07:53:40.527709891 (616h24m40.398792927s) [noise=true , address=77.90.31.12:4589 , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.528022527 (616h24m40.399105562s) [noise=true , address=45.91.116.6:4975 , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 31 07:53:40.528036814 (616h24m40.399119849s) [noise=true , address=45.91.116.6:4975 , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.528063705 (616h24m40.39914674s) [seg_pos={0 0}, upload=540182]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:53:40.528171600 (616h24m40.399254635s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:53:40.528177852 (616h24m40.399260887s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:53:40.824344511 (616h24m40.695427566s) [upload=540182]: Done uploading segments...
Dec 31 07:53:40.824370379 (616h24m40.695453454s) [upload=540182]: Upload finished. err: <nil>
Duration: 5.749972485s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X............X...........................................] duration:
43.032792031s trace:
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:56:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 43.03281196s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..............................X.........X............X..X...] duration:
3.38994864s trace:
https://jaeger.dev.storj.io/trace/e2308925a42e319
FAIL (2025-12-31 07:56:23 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.389978647s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X............X..........................................] duration:
16.951474178s trace:
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:55:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 16.951490018s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
3.568267625s trace:
https://jaeger.dev.storj.io/trace/59c3b821b75eba3a
FAIL (2025-12-31 07:55:53 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 3.568298625s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
6.921674072s trace:
https://jaeger.dev.storj.io/trace/5dd0609e3a2ac547
FAIL (2025-12-31 07:55:49 UTC)
orbiter warning: context deadline exceeded
Duration: 6.921695904s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...............................X.........X............X..X..] duration:
3.694858755s trace:
https://jaeger.dev.storj.io/trace/e2308925a42e319
FAIL (2025-12-31 07:55:47 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.694882149s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X............X.........................................] duration:
36.70785404s trace:
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:55:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 36.707872756s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.X.X.X.X.X......XXX.X....XXX..X...........X......X.X.X.XX...] duration:
5.999186069s trace:
https://jaeger.dev.storj.io/trace/6c9f60d6c2a165da
FAIL (2025-12-31 07:55:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA) context deadline exceeded
======LOGS======
Dec 31 07:53:38.027484676 (616h24m37.898567721s) [upload=540182]: Starting upload
Dec 31 07:53:38.027534512 (616h24m37.898617547s) [upload=540182]: Done starting upload
Dec 31 07:53:38.027801700 (616h24m37.898884735s) [upload=540182]: Uploading segments...
Dec 31 07:53:38.027979689 (616h24m37.899062734s) [upload=540182]: Got next segment. Inline: false
Dec 31 07:53:38.028073868 (616h24m37.899156903s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:53:38.028082775 (616h24m37.89916581s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:53:38.277891170 (616h24m38.148974205s) [seg_pos={0 0}, upload=540182]: Begin upload segment...
Dec 31 07:53:38.277968046 (616h24m38.149051081s) [seg_pos={0 0}, upload=540182]: Done begin upload segment.
Dec 31 07:53:38.278029704 (616h24m38.149112749s) [noise=true , address=45.91.116.6:4975 , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278042758 (616h24m38.149125794s) [noise=true , address=77.90.31.12:4589 , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278144813 (616h24m38.149227848s) [noise=true , address=145.79.227.50:4375 , piece_id=CXKJCYVE6SJOOVQWHH34MCITFOFEV27SP4OKAPXF4DJT3EXIHWGQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278830739 (616h24m38.149913784s) [noise=true , address=77.90.31.19:4842 , piece_id=PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279143605 (616h24m38.15022664s) [noise=true , address=77.90.31.15:4700 , piece_id=ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279489213 (616h24m38.150572258s) [noise=true , address=45.91.116.13:4154 , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279878865 (616h24m38.15096191s) [noise=true , address=145.79.227.26:4655 , piece_id=YHNBMQCH6QZRCTHVFDZRVHMXB2BP3MJ7B3RG3QIPBQ4PQW5JYB6A, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.280269218 (616h24m38.151352253s) [noise=true , address=145.79.227.37:4566 , piece_id=6E73Y5FUFAGQLU4IYUQNJ2J7XXXRJQGPM7KJRHMYQ3ENEBN76QKQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.280653881 (616h24m38.151736926s) [noise=true , address=145.79.227.24:4581 , piece_id=IKQBNNGA2QEVTMZZMGPMHLLIKLEDKZCL34X2DDIFZFHW2GBKLCXA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.281852182 (616h24m38.152935227s) [noise=true , address=145.79.227.16:4910 , piece_id=6L6QC3KSIG6CVDPLG55O6Q24L3QZ72GL3OTPAETHJG7I4Z4WMYKA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.281889793 (616h24m38.152972828s) [noise=true , address=145.79.227.45:4229 , piece_id=3EKCB2QETOCEMR2WCFYDZKVYT2R3CBOOKMUHXBMANKMOJD7H7IZA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.283093776 (616h24m38.154176821s) [noise=true , address=145.79.227.12:4767 , piece_id=GGEX2BKWC4URKRI34IAR7Z2KAP62FVRDM3ELUTNIH6WB4QN3JLRQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.283497055 (616h24m38.15458009s) [noise=true , address=145.79.227.23:4545 , piece_id=F5KYOLWB2IZRRCKVYFGHFZ23UZV7FUR2PCF2HO7GSHEMKIY5IL3Q, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.284318098 (616h24m38.155401133s) [noise=true , address=145.79.227.51:4410 , piece_id=VYULD4SFKPQRYZSOPM2PUQC4LUQAQ4WXRYVSOSRYCA5N5LFV4FAA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.285603535 (616h24m38.15668657s) [noise=true , address=145.79.227.52:4446 , piece_id=6FI2W5DULY5FFDFIFXLNSDUX7EEVDGB3WLWIEEZXFKNOR3AIOSIQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.286960870 (616h24m38.158043915s) [noise=true , address=145.79.227.18:4006 , piece_id=LFDSYTOQR42MSDEIRDUXGRNBUF3ONMCL6BJTMRNLNDHHNY5V7ZDQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:39.976475363 (616h24m39.847558408s) [upload=540182]: Next returned nil segment
Dec 31 07:53:39.976503577 (616h24m39.847586622s) [upload=540182]: Waiting for error group managing segments...
Dec 31 07:53:40.263089662 (616h24m40.134172697s) [noise=true , address=145.79.227.52:4446 , piece_id=6FI2W5DULY5FFDFIFXLNSDUX7EEVDGB3WLWIEEZXFKNOR3AIOSIQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.263514351 (616h24m40.134597386s) [noise=true , address=145.79.227.16:4910 , piece_id=6L6QC3KSIG6CVDPLG55O6Q24L3QZ72GL3OTPAETHJG7I4Z4WMYKA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.266215304 (616h24m40.137298339s) [noise=true , address=145.79.227.12:4767 , piece_id=GGEX2BKWC4URKRI34IAR7Z2KAP62FVRDM3ELUTNIH6WB4QN3JLRQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.274366505 (616h24m40.14544955s) [noise=true , address=145.79.227.51:4410 , piece_id=VYULD4SFKPQRYZSOPM2PUQC4LUQAQ4WXRYVSOSRYCA5N5LFV4FAA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.277241870 (616h24m40.148324905s) [noise=true , address=145.79.227.26:4655 , piece_id=YHNBMQCH6QZRCTHVFDZRVHMXB2BP3MJ7B3RG3QIPBQ4PQW5JYB6A, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.277655779 (616h24m40.148738814s) [noise=true , address=145.79.227.37:4566 , piece_id=6E73Y5FUFAGQLU4IYUQNJ2J7XXXRJQGPM7KJRHMYQ3ENEBN76QKQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.288485420 (616h24m40.159568465s) [noise=true , address=145.79.227.50:4375 , piece_id=CXKJCYVE6SJOOVQWHH34MCITFOFEV27SP4OKAPXF4DJT3EXIHWGQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.290211387 (616h24m40.161294432s) [noise=true , address=145.79.227.18:4006 , piece_id=LFDSYTOQR42MSDEIRDUXGRNBUF3ONMCL6BJTMRNLNDHHNY5V7ZDQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.293396742 (616h24m40.164479787s) [noise=true , address=145.79.227.24:4581 , piece_id=IKQBNNGA2QEVTMZZMGPMHLLIKLEDKZCL34X2DDIFZFHW2GBKLCXA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.302471963 (616h24m40.173554998s) [noise=true , address=145.79.227.45:4229 , piece_id=3EKCB2QETOCEMR2WCFYDZKVYT2R3CBOOKMUHXBMANKMOJD7H7IZA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.354700213 (616h24m40.225783248s) [noise=true , address=145.79.227.23:4545 , piece_id=F5KYOLWB2IZRRCKVYFGHFZ23UZV7FUR2PCF2HO7GSHEMKIY5IL3Q, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.453140008 (616h24m40.324223043s) [noise=true , address=77.90.31.19:4842 , piece_id=PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.527310551 (616h24m40.398393596s) [noise=true , address=77.90.31.15:4700 , piece_id=ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.527327724 (616h24m40.398410759s) [seg_pos={0 0}, upload=540182]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:53:40.527495442 (616h24m40.398578477s) [noise=true , address=45.91.116.13:4154 , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 31 07:53:40.527501184 (616h24m40.398584219s) [noise=true , address=45.91.116.13:4154 , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.527703939 (616h24m40.398786974s) [noise=true , address=77.90.31.12:4589 , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection
Dec 31 07:53:40.527709891 (616h24m40.398792927s) [noise=true , address=77.90.31.12:4589 , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.528022527 (616h24m40.399105562s) [noise=true , address=45.91.116.6:4975 , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 31 07:53:40.528036814 (616h24m40.399119849s) [noise=true , address=45.91.116.6:4975 , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.528063705 (616h24m40.39914674s) [seg_pos={0 0}, upload=540182]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:53:40.528171600 (616h24m40.399254635s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:53:40.528177852 (616h24m40.399260887s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:53:40.824344511 (616h24m40.695427566s) [upload=540182]: Done uploading segments...
Dec 31 07:53:40.824370379 (616h24m40.695453454s) [upload=540182]: Upload finished. err: <nil>
Duration: 5.999212781s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
6.808919399s trace:
https://jaeger.dev.storj.io/trace/5dd0609e3a2ac547
FAIL (2025-12-31 07:55:25 UTC)
orbiter warning: context deadline exceeded
Duration: 6.808941914s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [..X.X.X.X.X......XXX.X....XXX..X...........X......X.X.X.XX..] duration:
5.890387932s trace:
https://jaeger.dev.storj.io/trace/6c9f60d6c2a165da
FAIL (2025-12-31 07:55:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA) context deadline exceeded
======LOGS======
Dec 31 07:53:38.027484676 (616h24m37.898567721s) [upload=540182]: Starting upload
Dec 31 07:53:38.027534512 (616h24m37.898617547s) [upload=540182]: Done starting upload
Dec 31 07:53:38.027801700 (616h24m37.898884735s) [upload=540182]: Uploading segments...
Dec 31 07:53:38.027979689 (616h24m37.899062734s) [upload=540182]: Got next segment. Inline: false
Dec 31 07:53:38.028073868 (616h24m37.899156903s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:53:38.028082775 (616h24m37.89916581s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:53:38.277891170 (616h24m38.148974205s) [seg_pos={0 0}, upload=540182]: Begin upload segment...
Dec 31 07:53:38.277968046 (616h24m38.149051081s) [seg_pos={0 0}, upload=540182]: Done begin upload segment.
Dec 31 07:53:38.278029704 (616h24m38.149112749s) [noise=true , address=45.91.116.6:4975 , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278042758 (616h24m38.149125794s) [noise=true , address=77.90.31.12:4589 , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278144813 (616h24m38.149227848s) [noise=true , address=145.79.227.50:4375 , piece_id=CXKJCYVE6SJOOVQWHH34MCITFOFEV27SP4OKAPXF4DJT3EXIHWGQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278830739 (616h24m38.149913784s) [noise=true , address=77.90.31.19:4842 , piece_id=PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279143605 (616h24m38.15022664s) [noise=true , address=77.90.31.15:4700 , piece_id=ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279489213 (616h24m38.150572258s) [noise=true , address=45.91.116.13:4154 , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279878865 (616h24m38.15096191s) [noise=true , address=145.79.227.26:4655 , piece_id=YHNBMQCH6QZRCTHVFDZRVHMXB2BP3MJ7B3RG3QIPBQ4PQW5JYB6A, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.280269218 (616h24m38.151352253s) [noise=true , address=145.79.227.37:4566 , piece_id=6E73Y5FUFAGQLU4IYUQNJ2J7XXXRJQGPM7KJRHMYQ3ENEBN76QKQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.280653881 (616h24m38.151736926s) [noise=true , address=145.79.227.24:4581 , piece_id=IKQBNNGA2QEVTMZZMGPMHLLIKLEDKZCL34X2DDIFZFHW2GBKLCXA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.281852182 (616h24m38.152935227s) [noise=true , address=145.79.227.16:4910 , piece_id=6L6QC3KSIG6CVDPLG55O6Q24L3QZ72GL3OTPAETHJG7I4Z4WMYKA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.281889793 (616h24m38.152972828s) [noise=true , address=145.79.227.45:4229 , piece_id=3EKCB2QETOCEMR2WCFYDZKVYT2R3CBOOKMUHXBMANKMOJD7H7IZA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.283093776 (616h24m38.154176821s) [noise=true , address=145.79.227.12:4767 , piece_id=GGEX2BKWC4URKRI34IAR7Z2KAP62FVRDM3ELUTNIH6WB4QN3JLRQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.283497055 (616h24m38.15458009s) [noise=true , address=145.79.227.23:4545 , piece_id=F5KYOLWB2IZRRCKVYFGHFZ23UZV7FUR2PCF2HO7GSHEMKIY5IL3Q, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.284318098 (616h24m38.155401133s) [noise=true , address=145.79.227.51:4410 , piece_id=VYULD4SFKPQRYZSOPM2PUQC4LUQAQ4WXRYVSOSRYCA5N5LFV4FAA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.285603535 (616h24m38.15668657s) [noise=true , address=145.79.227.52:4446 , piece_id=6FI2W5DULY5FFDFIFXLNSDUX7EEVDGB3WLWIEEZXFKNOR3AIOSIQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.286960870 (616h24m38.158043915s) [noise=true , address=145.79.227.18:4006 , piece_id=LFDSYTOQR42MSDEIRDUXGRNBUF3ONMCL6BJTMRNLNDHHNY5V7ZDQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:39.976475363 (616h24m39.847558408s) [upload=540182]: Next returned nil segment
Dec 31 07:53:39.976503577 (616h24m39.847586622s) [upload=540182]: Waiting for error group managing segments...
Dec 31 07:53:40.263089662 (616h24m40.134172697s) [noise=true , address=145.79.227.52:4446 , piece_id=6FI2W5DULY5FFDFIFXLNSDUX7EEVDGB3WLWIEEZXFKNOR3AIOSIQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.263514351 (616h24m40.134597386s) [noise=true , address=145.79.227.16:4910 , piece_id=6L6QC3KSIG6CVDPLG55O6Q24L3QZ72GL3OTPAETHJG7I4Z4WMYKA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.266215304 (616h24m40.137298339s) [noise=true , address=145.79.227.12:4767 , piece_id=GGEX2BKWC4URKRI34IAR7Z2KAP62FVRDM3ELUTNIH6WB4QN3JLRQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.274366505 (616h24m40.14544955s) [noise=true , address=145.79.227.51:4410 , piece_id=VYULD4SFKPQRYZSOPM2PUQC4LUQAQ4WXRYVSOSRYCA5N5LFV4FAA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.277241870 (616h24m40.148324905s) [noise=true , address=145.79.227.26:4655 , piece_id=YHNBMQCH6QZRCTHVFDZRVHMXB2BP3MJ7B3RG3QIPBQ4PQW5JYB6A, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.277655779 (616h24m40.148738814s) [noise=true , address=145.79.227.37:4566 , piece_id=6E73Y5FUFAGQLU4IYUQNJ2J7XXXRJQGPM7KJRHMYQ3ENEBN76QKQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.288485420 (616h24m40.159568465s) [noise=true , address=145.79.227.50:4375 , piece_id=CXKJCYVE6SJOOVQWHH34MCITFOFEV27SP4OKAPXF4DJT3EXIHWGQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.290211387 (616h24m40.161294432s) [noise=true , address=145.79.227.18:4006 , piece_id=LFDSYTOQR42MSDEIRDUXGRNBUF3ONMCL6BJTMRNLNDHHNY5V7ZDQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.293396742 (616h24m40.164479787s) [noise=true , address=145.79.227.24:4581 , piece_id=IKQBNNGA2QEVTMZZMGPMHLLIKLEDKZCL34X2DDIFZFHW2GBKLCXA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.302471963 (616h24m40.173554998s) [noise=true , address=145.79.227.45:4229 , piece_id=3EKCB2QETOCEMR2WCFYDZKVYT2R3CBOOKMUHXBMANKMOJD7H7IZA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.354700213 (616h24m40.225783248s) [noise=true , address=145.79.227.23:4545 , piece_id=F5KYOLWB2IZRRCKVYFGHFZ23UZV7FUR2PCF2HO7GSHEMKIY5IL3Q, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.453140008 (616h24m40.324223043s) [noise=true , address=77.90.31.19:4842 , piece_id=PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.527310551 (616h24m40.398393596s) [noise=true , address=77.90.31.15:4700 , piece_id=ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.527327724 (616h24m40.398410759s) [seg_pos={0 0}, upload=540182]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:53:40.527495442 (616h24m40.398578477s) [noise=true , address=45.91.116.13:4154 , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 31 07:53:40.527501184 (616h24m40.398584219s) [noise=true , address=45.91.116.13:4154 , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.527703939 (616h24m40.398786974s) [noise=true , address=77.90.31.12:4589 , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection
Dec 31 07:53:40.527709891 (616h24m40.398792927s) [noise=true , address=77.90.31.12:4589 , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.528022527 (616h24m40.399105562s) [noise=true , address=45.91.116.6:4975 , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 31 07:53:40.528036814 (616h24m40.399119849s) [noise=true , address=45.91.116.6:4975 , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.528063705 (616h24m40.39914674s) [seg_pos={0 0}, upload=540182]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:53:40.528171600 (616h24m40.399254635s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:53:40.528177852 (616h24m40.399260887s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:53:40.824344511 (616h24m40.695427566s) [upload=540182]: Done uploading segments...
Dec 31 07:53:40.824370379 (616h24m40.695453454s) [upload=540182]: Upload finished. err: <nil>
Duration: 5.890416497s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [................................X.........X............X..X.] duration:
3.536982714s trace:
https://jaeger.dev.storj.io/trace/e2308925a42e319
FAIL (2025-12-31 07:55:14 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.537018552s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X............X........................................] duration:
24.597363932s trace:
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:55:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 24.597397576s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
6.137251226s trace:
https://jaeger.dev.storj.io/trace/5dd0609e3a2ac547
FAIL (2025-12-31 07:54:44 UTC)
orbiter warning: context deadline exceeded
Duration: 6.137276975s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
2.606314865s trace:
https://jaeger.dev.storj.io/trace/59c3b821b75eba3a
FAIL (2025-12-31 07:54:38 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 2.60634875s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X............X.......................................] duration:
29.383483901s trace:
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:54:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 29.383500623s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [...X.X.X.X.X......XXX.X....XXX..X...........X......X.X.X.XX.] duration:
6.115880242s trace:
https://jaeger.dev.storj.io/trace/6c9f60d6c2a165da
FAIL (2025-12-31 07:54:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA) context deadline exceeded
======LOGS======
Dec 31 07:53:38.027484676 (616h24m37.898567721s) [upload=540182]: Starting upload
Dec 31 07:53:38.027534512 (616h24m37.898617547s) [upload=540182]: Done starting upload
Dec 31 07:53:38.027801700 (616h24m37.898884735s) [upload=540182]: Uploading segments...
Dec 31 07:53:38.027979689 (616h24m37.899062734s) [upload=540182]: Got next segment. Inline: false
Dec 31 07:53:38.028073868 (616h24m37.899156903s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:53:38.028082775 (616h24m37.89916581s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:53:38.277891170 (616h24m38.148974205s) [seg_pos={0 0}, upload=540182]: Begin upload segment...
Dec 31 07:53:38.277968046 (616h24m38.149051081s) [seg_pos={0 0}, upload=540182]: Done begin upload segment.
Dec 31 07:53:38.278029704 (616h24m38.149112749s) [noise=true , address=45.91.116.6:4975 , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278042758 (616h24m38.149125794s) [noise=true , address=77.90.31.12:4589 , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278144813 (616h24m38.149227848s) [noise=true , address=145.79.227.50:4375 , piece_id=CXKJCYVE6SJOOVQWHH34MCITFOFEV27SP4OKAPXF4DJT3EXIHWGQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278830739 (616h24m38.149913784s) [noise=true , address=77.90.31.19:4842 , piece_id=PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279143605 (616h24m38.15022664s) [noise=true , address=77.90.31.15:4700 , piece_id=ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279489213 (616h24m38.150572258s) [noise=true , address=45.91.116.13:4154 , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279878865 (616h24m38.15096191s) [noise=true , address=145.79.227.26:4655 , piece_id=YHNBMQCH6QZRCTHVFDZRVHMXB2BP3MJ7B3RG3QIPBQ4PQW5JYB6A, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.280269218 (616h24m38.151352253s) [noise=true , address=145.79.227.37:4566 , piece_id=6E73Y5FUFAGQLU4IYUQNJ2J7XXXRJQGPM7KJRHMYQ3ENEBN76QKQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.280653881 (616h24m38.151736926s) [noise=true , address=145.79.227.24:4581 , piece_id=IKQBNNGA2QEVTMZZMGPMHLLIKLEDKZCL34X2DDIFZFHW2GBKLCXA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.281852182 (616h24m38.152935227s) [noise=true , address=145.79.227.16:4910 , piece_id=6L6QC3KSIG6CVDPLG55O6Q24L3QZ72GL3OTPAETHJG7I4Z4WMYKA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.281889793 (616h24m38.152972828s) [noise=true , address=145.79.227.45:4229 , piece_id=3EKCB2QETOCEMR2WCFYDZKVYT2R3CBOOKMUHXBMANKMOJD7H7IZA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.283093776 (616h24m38.154176821s) [noise=true , address=145.79.227.12:4767 , piece_id=GGEX2BKWC4URKRI34IAR7Z2KAP62FVRDM3ELUTNIH6WB4QN3JLRQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.283497055 (616h24m38.15458009s) [noise=true , address=145.79.227.23:4545 , piece_id=F5KYOLWB2IZRRCKVYFGHFZ23UZV7FUR2PCF2HO7GSHEMKIY5IL3Q, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.284318098 (616h24m38.155401133s) [noise=true , address=145.79.227.51:4410 , piece_id=VYULD4SFKPQRYZSOPM2PUQC4LUQAQ4WXRYVSOSRYCA5N5LFV4FAA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.285603535 (616h24m38.15668657s) [noise=true , address=145.79.227.52:4446 , piece_id=6FI2W5DULY5FFDFIFXLNSDUX7EEVDGB3WLWIEEZXFKNOR3AIOSIQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.286960870 (616h24m38.158043915s) [noise=true , address=145.79.227.18:4006 , piece_id=LFDSYTOQR42MSDEIRDUXGRNBUF3ONMCL6BJTMRNLNDHHNY5V7ZDQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:39.976475363 (616h24m39.847558408s) [upload=540182]: Next returned nil segment
Dec 31 07:53:39.976503577 (616h24m39.847586622s) [upload=540182]: Waiting for error group managing segments...
Dec 31 07:53:40.263089662 (616h24m40.134172697s) [noise=true , address=145.79.227.52:4446 , piece_id=6FI2W5DULY5FFDFIFXLNSDUX7EEVDGB3WLWIEEZXFKNOR3AIOSIQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.263514351 (616h24m40.134597386s) [noise=true , address=145.79.227.16:4910 , piece_id=6L6QC3KSIG6CVDPLG55O6Q24L3QZ72GL3OTPAETHJG7I4Z4WMYKA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.266215304 (616h24m40.137298339s) [noise=true , address=145.79.227.12:4767 , piece_id=GGEX2BKWC4URKRI34IAR7Z2KAP62FVRDM3ELUTNIH6WB4QN3JLRQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.274366505 (616h24m40.14544955s) [noise=true , address=145.79.227.51:4410 , piece_id=VYULD4SFKPQRYZSOPM2PUQC4LUQAQ4WXRYVSOSRYCA5N5LFV4FAA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.277241870 (616h24m40.148324905s) [noise=true , address=145.79.227.26:4655 , piece_id=YHNBMQCH6QZRCTHVFDZRVHMXB2BP3MJ7B3RG3QIPBQ4PQW5JYB6A, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.277655779 (616h24m40.148738814s) [noise=true , address=145.79.227.37:4566 , piece_id=6E73Y5FUFAGQLU4IYUQNJ2J7XXXRJQGPM7KJRHMYQ3ENEBN76QKQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.288485420 (616h24m40.159568465s) [noise=true , address=145.79.227.50:4375 , piece_id=CXKJCYVE6SJOOVQWHH34MCITFOFEV27SP4OKAPXF4DJT3EXIHWGQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.290211387 (616h24m40.161294432s) [noise=true , address=145.79.227.18:4006 , piece_id=LFDSYTOQR42MSDEIRDUXGRNBUF3ONMCL6BJTMRNLNDHHNY5V7ZDQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.293396742 (616h24m40.164479787s) [noise=true , address=145.79.227.24:4581 , piece_id=IKQBNNGA2QEVTMZZMGPMHLLIKLEDKZCL34X2DDIFZFHW2GBKLCXA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.302471963 (616h24m40.173554998s) [noise=true , address=145.79.227.45:4229 , piece_id=3EKCB2QETOCEMR2WCFYDZKVYT2R3CBOOKMUHXBMANKMOJD7H7IZA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.354700213 (616h24m40.225783248s) [noise=true , address=145.79.227.23:4545 , piece_id=F5KYOLWB2IZRRCKVYFGHFZ23UZV7FUR2PCF2HO7GSHEMKIY5IL3Q, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.453140008 (616h24m40.324223043s) [noise=true , address=77.90.31.19:4842 , piece_id=PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.527310551 (616h24m40.398393596s) [noise=true , address=77.90.31.15:4700 , piece_id=ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.527327724 (616h24m40.398410759s) [seg_pos={0 0}, upload=540182]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:53:40.527495442 (616h24m40.398578477s) [noise=true , address=45.91.116.13:4154 , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 31 07:53:40.527501184 (616h24m40.398584219s) [noise=true , address=45.91.116.13:4154 , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.527703939 (616h24m40.398786974s) [noise=true , address=77.90.31.12:4589 , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection
Dec 31 07:53:40.527709891 (616h24m40.398792927s) [noise=true , address=77.90.31.12:4589 , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.528022527 (616h24m40.399105562s) [noise=true , address=45.91.116.6:4975 , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 31 07:53:40.528036814 (616h24m40.399119849s) [noise=true , address=45.91.116.6:4975 , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.528063705 (616h24m40.39914674s) [seg_pos={0 0}, upload=540182]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:53:40.528171600 (616h24m40.399254635s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:53:40.528177852 (616h24m40.399260887s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:53:40.824344511 (616h24m40.695427566s) [upload=540182]: Done uploading segments...
Dec 31 07:53:40.824370379 (616h24m40.695453454s) [upload=540182]: Upload finished. err: <nil>
Duration: 6.115895953s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [X................................X.........X............X..X] duration:
5.001432114s trace:
https://jaeger.dev.storj.io/trace/e2308925a42e319
FAIL (2025-12-31 07:54:08 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001453675s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [....X.X.X.X.X......XXX.X....XXX..X...........X......X.X.X.XX] duration:
30.003931798s trace:
https://jaeger.dev.storj.io/trace/6c9f60d6c2a165da
FAIL (2025-12-31 07:54:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA) context deadline exceeded
======LOGS======
Dec 31 07:53:38.027484676 (616h24m37.898567721s) [upload=540182]: Starting upload
Dec 31 07:53:38.027534512 (616h24m37.898617547s) [upload=540182]: Done starting upload
Dec 31 07:53:38.027801700 (616h24m37.898884735s) [upload=540182]: Uploading segments...
Dec 31 07:53:38.027979689 (616h24m37.899062734s) [upload=540182]: Got next segment. Inline: false
Dec 31 07:53:38.028073868 (616h24m37.899156903s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:53:38.028082775 (616h24m37.89916581s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:53:38.277891170 (616h24m38.148974205s) [seg_pos={0 0}, upload=540182]: Begin upload segment...
Dec 31 07:53:38.277968046 (616h24m38.149051081s) [seg_pos={0 0}, upload=540182]: Done begin upload segment.
Dec 31 07:53:38.278029704 (616h24m38.149112749s) [noise=true , address=45.91.116.6:4975 , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278042758 (616h24m38.149125794s) [noise=true , address=77.90.31.12:4589 , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278144813 (616h24m38.149227848s) [noise=true , address=145.79.227.50:4375 , piece_id=CXKJCYVE6SJOOVQWHH34MCITFOFEV27SP4OKAPXF4DJT3EXIHWGQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278830739 (616h24m38.149913784s) [noise=true , address=77.90.31.19:4842 , piece_id=PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279143605 (616h24m38.15022664s) [noise=true , address=77.90.31.15:4700 , piece_id=ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279489213 (616h24m38.150572258s) [noise=true , address=45.91.116.13:4154 , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279878865 (616h24m38.15096191s) [noise=true , address=145.79.227.26:4655 , piece_id=YHNBMQCH6QZRCTHVFDZRVHMXB2BP3MJ7B3RG3QIPBQ4PQW5JYB6A, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.280269218 (616h24m38.151352253s) [noise=true , address=145.79.227.37:4566 , piece_id=6E73Y5FUFAGQLU4IYUQNJ2J7XXXRJQGPM7KJRHMYQ3ENEBN76QKQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.280653881 (616h24m38.151736926s) [noise=true , address=145.79.227.24:4581 , piece_id=IKQBNNGA2QEVTMZZMGPMHLLIKLEDKZCL34X2DDIFZFHW2GBKLCXA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.281852182 (616h24m38.152935227s) [noise=true , address=145.79.227.16:4910 , piece_id=6L6QC3KSIG6CVDPLG55O6Q24L3QZ72GL3OTPAETHJG7I4Z4WMYKA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.281889793 (616h24m38.152972828s) [noise=true , address=145.79.227.45:4229 , piece_id=3EKCB2QETOCEMR2WCFYDZKVYT2R3CBOOKMUHXBMANKMOJD7H7IZA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.283093776 (616h24m38.154176821s) [noise=true , address=145.79.227.12:4767 , piece_id=GGEX2BKWC4URKRI34IAR7Z2KAP62FVRDM3ELUTNIH6WB4QN3JLRQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.283497055 (616h24m38.15458009s) [noise=true , address=145.79.227.23:4545 , piece_id=F5KYOLWB2IZRRCKVYFGHFZ23UZV7FUR2PCF2HO7GSHEMKIY5IL3Q, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.284318098 (616h24m38.155401133s) [noise=true , address=145.79.227.51:4410 , piece_id=VYULD4SFKPQRYZSOPM2PUQC4LUQAQ4WXRYVSOSRYCA5N5LFV4FAA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.285603535 (616h24m38.15668657s) [noise=true , address=145.79.227.52:4446 , piece_id=6FI2W5DULY5FFDFIFXLNSDUX7EEVDGB3WLWIEEZXFKNOR3AIOSIQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.286960870 (616h24m38.158043915s) [noise=true , address=145.79.227.18:4006 , piece_id=LFDSYTOQR42MSDEIRDUXGRNBUF3ONMCL6BJTMRNLNDHHNY5V7ZDQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:39.976475363 (616h24m39.847558408s) [upload=540182]: Next returned nil segment
Dec 31 07:53:39.976503577 (616h24m39.847586622s) [upload=540182]: Waiting for error group managing segments...
Dec 31 07:53:40.263089662 (616h24m40.134172697s) [noise=true , address=145.79.227.52:4446 , piece_id=6FI2W5DULY5FFDFIFXLNSDUX7EEVDGB3WLWIEEZXFKNOR3AIOSIQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.263514351 (616h24m40.134597386s) [noise=true , address=145.79.227.16:4910 , piece_id=6L6QC3KSIG6CVDPLG55O6Q24L3QZ72GL3OTPAETHJG7I4Z4WMYKA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.266215304 (616h24m40.137298339s) [noise=true , address=145.79.227.12:4767 , piece_id=GGEX2BKWC4URKRI34IAR7Z2KAP62FVRDM3ELUTNIH6WB4QN3JLRQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.274366505 (616h24m40.14544955s) [noise=true , address=145.79.227.51:4410 , piece_id=VYULD4SFKPQRYZSOPM2PUQC4LUQAQ4WXRYVSOSRYCA5N5LFV4FAA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.277241870 (616h24m40.148324905s) [noise=true , address=145.79.227.26:4655 , piece_id=YHNBMQCH6QZRCTHVFDZRVHMXB2BP3MJ7B3RG3QIPBQ4PQW5JYB6A, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.277655779 (616h24m40.148738814s) [noise=true , address=145.79.227.37:4566 , piece_id=6E73Y5FUFAGQLU4IYUQNJ2J7XXXRJQGPM7KJRHMYQ3ENEBN76QKQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.288485420 (616h24m40.159568465s) [noise=true , address=145.79.227.50:4375 , piece_id=CXKJCYVE6SJOOVQWHH34MCITFOFEV27SP4OKAPXF4DJT3EXIHWGQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.290211387 (616h24m40.161294432s) [noise=true , address=145.79.227.18:4006 , piece_id=LFDSYTOQR42MSDEIRDUXGRNBUF3ONMCL6BJTMRNLNDHHNY5V7ZDQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.293396742 (616h24m40.164479787s) [noise=true , address=145.79.227.24:4581 , piece_id=IKQBNNGA2QEVTMZZMGPMHLLIKLEDKZCL34X2DDIFZFHW2GBKLCXA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.302471963 (616h24m40.173554998s) [noise=true , address=145.79.227.45:4229 , piece_id=3EKCB2QETOCEMR2WCFYDZKVYT2R3CBOOKMUHXBMANKMOJD7H7IZA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.354700213 (616h24m40.225783248s) [noise=true , address=145.79.227.23:4545 , piece_id=F5KYOLWB2IZRRCKVYFGHFZ23UZV7FUR2PCF2HO7GSHEMKIY5IL3Q, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.453140008 (616h24m40.324223043s) [noise=true , address=77.90.31.19:4842 , piece_id=PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.527310551 (616h24m40.398393596s) [noise=true , address=77.90.31.15:4700 , piece_id=ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.527327724 (616h24m40.398410759s) [seg_pos={0 0}, upload=540182]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:53:40.527495442 (616h24m40.398578477s) [noise=true , address=45.91.116.13:4154 , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 31 07:53:40.527501184 (616h24m40.398584219s) [noise=true , address=45.91.116.13:4154 , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.527703939 (616h24m40.398786974s) [noise=true , address=77.90.31.12:4589 , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection
Dec 31 07:53:40.527709891 (616h24m40.398792927s) [noise=true , address=77.90.31.12:4589 , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.528022527 (616h24m40.399105562s) [noise=true , address=45.91.116.6:4975 , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 31 07:53:40.528036814 (616h24m40.399119849s) [noise=true , address=45.91.116.6:4975 , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.528063705 (616h24m40.39914674s) [seg_pos={0 0}, upload=540182]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:53:40.528171600 (616h24m40.399254635s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:53:40.528177852 (616h24m40.399260887s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:53:40.824344511 (616h24m40.695427566s) [upload=540182]: Done uploading segments...
Dec 31 07:53:40.824370379 (616h24m40.695453454s) [upload=540182]: Upload finished. err: <nil>
Duration: 30.003945965s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
2.470161477s trace:
https://jaeger.dev.storj.io/trace/59c3b821b75eba3a
FAIL (2025-12-31 07:53:55 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 2.47018871s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
16.516682423s trace:
https://jaeger.dev.storj.io/trace/5dd0609e3a2ac547
FAIL (2025-12-31 07:53:50 UTC)
orbiter warning: context deadline exceeded
Duration: 16.516702902s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [........X............X......................................] duration:
41.890720569s trace:
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:53:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 41.89075255s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
3.219385651s trace:
https://jaeger.dev.storj.io/trace/59c3b821b75eba3a
FAIL (2025-12-31 07:53:38 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 3.219435176s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
3.586716175s trace:
https://jaeger.dev.storj.io/trace/59c3b821b75eba3a
FAIL (2025-12-31 07:53:07 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 3.586733018s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X....X.X.X.X.X......XXX.X....XXX..X...........X......X.X.X.X] duration:
30.003479253s trace:
https://jaeger.dev.storj.io/trace/780f2a5d425cca30
FAIL (2025-12-31 07:52:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 2UJBHMT3INMNVGMJRLAE5GRJZX2Z7QU4VANEYZZUFBVW5MQJHU7Q) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: SY3FY6XNHIYZNGAIIH4MFZTYWAJ7742Y7DTPVQ7YGFU5OBI6AXIQ) context deadline exceeded
======LOGS======
t segment. Inline: false
Dec 31 07:52:23.825346530 (616h23m23.696429575s) [upload=540161]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:52:23.825353353 (616h23m23.696436408s) [upload=540161]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:52:24.074620447 (616h23m23.945703482s) [seg_pos={0 0}, upload=540161]: Begin upload segment...
Dec 31 07:52:24.074671584 (616h23m23.945754629s) [seg_pos={0 0}, upload=540161]: Done begin upload segment.
Dec 31 07:52:24.074706801 (616h23m23.945789846s) [noise=true , address=145.79.227.23:4546 , piece_id=7ZDKTSRFXS3MA3LASWFHJAH4PKBQR6KCXPOCF67PE5L67HAHR22A, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.074842459 (616h23m23.945925504s) [noise=true , address=145.79.227.16:4910 , piece_id=B7SVVOHQGMLHW2MRYXPVGM7FRJVA24VA36EFOMRZ5455NUUQO2MQ, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.075343253 (616h23m23.946426298s) [noise=true , address=145.79.227.41:4086 , piece_id=Z4EACFPQMDUKTECKPNZJWSPY2TWOPA34762C5A444LOQSAEETUKA, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.076077171 (616h23m23.947160216s) [noise=true , address=145.79.227.19:4077 , piece_id=6OPSEG3UJICOOO3FB7C5PUDJABWHJ6A675P3LOBKKLXXQSBQ4K5Q, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.076233398 (616h23m23.947316443s) [noise=true , address=45.91.116.6:4975 , piece_id=SY3FY6XNHIYZNGAIIH4MFZTYWAJ7742Y7DTPVQ7YGFU5OBI6AXIQ, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.077034253 (616h23m23.948117288s) [noise=true , address=77.90.31.11:4554 , piece_id=KQ2MICSQHJWDJIJV3A34AG5LQBQ72NGXHDH5MTHGFNXMB2E53AYA, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.077068127 (616h23m23.948151162s) [noise=true , address=145.79.227.46:4603 , piece_id=WQXPW3ZMGWNSMCKKCUQCTPZEIVZODH2D2PT6BQEUTHLKEJL5LQVA, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.078082028 (616h23m23.949165073s) [noise=true , address=45.91.116.5:4300 , piece_id=ZLMSRRIIWQJPE6O7W7VE537CSSFXTCSAUAIQLYXMFP7LVUDC3AXQ, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.078723069 (616h23m23.949806114s) [noise=true , address=145.79.227.39:4014 , piece_id=MRV5LXPT44YDINXDSXB2YLP34VVQTXGXOMSB6E7VXVWHLCGNHHKQ, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.079269931 (616h23m23.950352966s) [noise=true , address=145.79.227.8:4329 , piece_id=B6OJP2STGAZ2MMZZ6Q6XS4LETPSIQFCHW32Q33DVRCSS3EILXAWA, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.079733483 (616h23m23.950816528s) [noise=true , address=145.79.227.42:4122 , piece_id=O6PSEQ7QUWP5MZXC4T3IEOUKXMZK6SME6DJBDW5IYIAIV2D7STAQ, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.080171987 (616h23m23.951255032s) [noise=true , address=145.79.227.24:4583 , piece_id=LGKDSJP424LWHOFRTWJBVPEY5XURZOBIQJGKISABZLRNACMHOP2A, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.081184636 (616h23m23.952267672s) [noise=true , address=45.91.116.14:4192 , piece_id=PA5XI2SEPTAM6NHLSZQU7WG5YJCTSONTMT5GGHNBIKBPTTXUVDQQ, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.081721027 (616h23m23.952804063s) [noise=true , address=45.91.116.10:4081 , piece_id=2UJBHMT3INMNVGMJRLAE5GRJZX2Z7QU4VANEYZZUFBVW5MQJHU7Q, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.082054523 (616h23m23.953137568s) [noise=true , address=45.91.116.7:4012 , piece_id=ZEHO5WPKHJCC3PDYJ24CQ6IRWV2MZHHBZGEGIPXDUN2THXI3XPTQ, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.082268981 (616h23m23.953352026s) [noise=true , address=145.79.227.50:4375 , piece_id=GQS7PTQTWLEKXJCOKXN6HZCL4DDUV2R6M662WLFLV64YMLB3OLNA, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:25.682650871 (616h23m25.553733906s) [upload=540161]: Next returned nil segment
Dec 31 07:52:25.682662573 (616h23m25.553745618s) [upload=540161]: Waiting for error group managing segments...
Dec 31 07:52:26.064740170 (616h23m25.935823215s) [noise=true , address=145.79.227.42:4122 , piece_id=O6PSEQ7QUWP5MZXC4T3IEOUKXMZK6SME6DJBDW5IYIAIV2D7STAQ, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.065650974 (616h23m25.936734009s) [noise=true , address=145.79.227.23:4546 , piece_id=7ZDKTSRFXS3MA3LASWFHJAH4PKBQR6KCXPOCF67PE5L67HAHR22A, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.068003122 (616h23m25.939086157s) [noise=true , address=145.79.227.19:4077 , piece_id=6OPSEG3UJICOOO3FB7C5PUDJABWHJ6A675P3LOBKKLXXQSBQ4K5Q, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.068771295 (616h23m25.93985433s) [noise=true , address=145.79.227.39:4014 , piece_id=MRV5LXPT44YDINXDSXB2YLP34VVQTXGXOMSB6E7VXVWHLCGNHHKQ, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.082898245 (616h23m25.95398129s) [noise=true , address=145.79.227.41:4086 , piece_id=Z4EACFPQMDUKTECKPNZJWSPY2TWOPA34762C5A444LOQSAEETUKA, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.084216175 (616h23m25.95529922s) [noise=true , address=145.79.227.8:4329 , piece_id=B6OJP2STGAZ2MMZZ6Q6XS4LETPSIQFCHW32Q33DVRCSS3EILXAWA, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.091993925 (616h23m25.96307697s) [noise=true , address=145.79.227.46:4603 , piece_id=WQXPW3ZMGWNSMCKKCUQCTPZEIVZODH2D2PT6BQEUTHLKEJL5LQVA, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.145871666 (616h23m26.016954701s) [noise=true , address=145.79.227.16:4910 , piece_id=B7SVVOHQGMLHW2MRYXPVGM7FRJVA24VA36EFOMRZ5455NUUQO2MQ, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.154342947 (616h23m26.025425982s) [noise=true , address=145.79.227.50:4375 , piece_id=GQS7PTQTWLEKXJCOKXN6HZCL4DDUV2R6M662WLFLV64YMLB3OLNA, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.156875870 (616h23m26.027958915s) [noise=true , address=145.79.227.24:4583 , piece_id=LGKDSJP424LWHOFRTWJBVPEY5XURZOBIQJGKISABZLRNACMHOP2A, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.404838770 (616h23m26.275921805s) [noise=true , address=45.91.116.6:4975 , piece_id=SY3FY6XNHIYZNGAIIH4MFZTYWAJ7742Y7DTPVQ7YGFU5OBI6AXIQ, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.487485923 (616h23m26.358568968s) [noise=true , address=45.91.116.10:4081 , piece_id=2UJBHMT3INMNVGMJRLAE5GRJZX2Z7QU4VANEYZZUFBVW5MQJHU7Q, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.558106627 (616h23m26.429189662s) [noise=true , address=45.91.116.14:4192 , piece_id=PA5XI2SEPTAM6NHLSZQU7WG5YJCTSONTMT5GGHNBIKBPTTXUVDQQ, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.558129981 (616h23m26.429213027s) [seg_pos={0 0}, upload=540161]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:52:26.558482913 (616h23m26.429565948s) [noise=true , address=45.91.116.7:4012 , piece_id=ZEHO5WPKHJCC3PDYJ24CQ6IRWV2MZHHBZGEGIPXDUN2THXI3XPTQ, seg_pos={0 0}, upload=540161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:46362->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:46362->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46362->45.91.116.7:4012: use of closed network connection
Dec 31 07:52:26.558491019 (616h23m26.429574064s) [noise=true , address=45.91.116.7:4012 , piece_id=ZEHO5WPKHJCC3PDYJ24CQ6IRWV2MZHHBZGEGIPXDUN2THXI3XPTQ, seg_pos={0 0}, upload=540161]: Long tail cancellation due to optimal threshold
Dec 31 07:52:26.558809525 (616h23m26.42989256s) [noise=true , address=77.90.31.11:4554 , piece_id=KQ2MICSQHJWDJIJV3A34AG5LQBQ72NGXHDH5MTHGFNXMB2E53AYA, seg_pos={0 0}, upload=540161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:44202->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:44202->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->77.90.31.11:4554: use of closed network connection
Dec 31 07:52:26.558818873 (616h23m26.429901919s) [noise=true , address=77.90.31.11:4554 , piece_id=KQ2MICSQHJWDJIJV3A34AG5LQBQ72NGXHDH5MTHGFNXMB2E53AYA, seg_pos={0 0}, upload=540161]: Long tail cancellation due to optimal threshold
Dec 31 07:52:26.559143732 (616h23m26.430226767s) [noise=true , address=45.91.116.5:4300 , piece_id=ZLMSRRIIWQJPE6O7W7VE537CSSFXTCSAUAIQLYXMFP7LVUDC3AXQ, seg_pos={0 0}, upload=540161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): internal: context canceled; piecestore: piecestore close: write tcp 188.166.225.170:60356->45.91.116.5:4300: use of closed network connection
Dec 31 07:52:26.559151727 (616h23m26.430234762s) [noise=true , address=45.91.116.5:4300 , piece_id=ZLMSRRIIWQJPE6O7W7VE537CSSFXTCSAUAIQLYXMFP7LVUDC3AXQ, seg_pos={0 0}, upload=540161]: Long tail cancellation due to optimal threshold
Dec 31 07:52:26.559170673 (616h23m26.430253718s) [seg_pos={0 0}, upload=540161]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:52:26.559276565 (616h23m26.4303596s) [upload=540161]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:52:26.559283297 (616h23m26.430366343s) [upload=540161]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:52:26.851054996 (616h23m26.722138032s) [upload=540161]: Done uploading segments...
Dec 31 07:52:26.851068904 (616h23m26.722151939s) [upload=540161]: Upload finished. err: <nil>
Duration: 30.003498259s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........X............X.....................................] duration:
15.175449843s trace:
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:52:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 15.175467177s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.X................................X.........X............X..] duration:
3.504088449s trace:
https://jaeger.dev.storj.io/trace/4dbc5f818d3c3b7
FAIL (2025-12-31 07:52:52 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.50410956s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........X............X....................................] duration:
47.965407746s trace:
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:52:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 47.965428094s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
6.850578263s trace:
https://jaeger.dev.storj.io/trace/5dd0609e3a2ac547
FAIL (2025-12-31 07:52:36 UTC)
orbiter warning: context deadline exceeded
Duration: 6.850600445s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
3.82334768s trace:
https://jaeger.dev.storj.io/trace/59c3b821b75eba3a
FAIL (2025-12-31 07:52:16 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 3.823385851s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..X................................X.........X............X.] duration:
4.689519597s trace:
https://jaeger.dev.storj.io/trace/4dbc5f818d3c3b7
FAIL (2025-12-31 07:52:02 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.689545786s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.X....X.X.X.X.X......XXX.X....XXX..X...........X......X.X.X.] duration:
5.828839369s trace:
https://jaeger.dev.storj.io/trace/b9a66a0708d887e
FAIL (2025-12-31 07:52:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 07:51:02.386794824 (616h22m2.257877859s) [upload=540141]: Starting upload
Dec 31 07:51:02.386883673 (616h22m2.257966748s) [upload=540141]: Done starting upload
Dec 31 07:51:02.387061752 (616h22m2.258144807s) [upload=540141]: Uploading segments...
Dec 31 07:51:02.387169487 (616h22m2.258252532s) [upload=540141]: Got next segment. Inline: false
Dec 31 07:51:02.387235102 (616h22m2.258318157s) [upload=540141]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:51:02.387242135 (616h22m2.25832518s) [upload=540141]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:51:02.631477047 (616h22m2.502560102s) [seg_pos={0 0}, upload=540141]: Begin upload segment...
Dec 31 07:51:02.631536671 (616h22m2.502619716s) [seg_pos={0 0}, upload=540141]: Done begin upload segment.
Dec 31 07:51:02.631572469 (616h22m2.502655524s) [noise=true , address=145.79.227.5:4223 , piece_id=U7A2ZGU3CUSID72JOCL23CVC3BSYOYHRW6MTDHGFMQB2V34MY7XA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.633123201 (616h22m2.504206246s) [noise=true , address=145.79.227.44:4193 , piece_id=AIE6JX7EQMVED7XDIWNQ65TZJTJVCY5CR7IWQCE37I3M6RYWVSKQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.634110732 (616h22m2.505193777s) [noise=true , address=145.79.227.36:4517 , piece_id=4GCERGAIUOSKTH533BBCFGSZUZLTOIJJWAMTFEKUMBZRQARYIK6Q, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.634670077 (616h22m2.505753122s) [noise=true , address=145.79.227.29:4943 , piece_id=7ZCXKYQET76RXQ4HVQJFLZBUSXCVVYYQO7B7VH5EFQX5RBFLD4QA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.635257716 (616h22m2.506340761s) [noise=true , address=77.90.31.16:4735 , piece_id=GPHX2PRAK34RGV4QSKDA2EP377FG26VY4RQAEYILTXVNXKYQ4IRA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.636355436 (616h22m2.507438471s) [noise=true , address=145.79.227.24:4582 , piece_id=QSMFJVHVRFVFDL4DT2UHLWMVF66HTQQ3X3HETXKVL4JERZFCP4QA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.637031744 (616h22m2.508114789s) [noise=true , address=145.79.227.13:4804 , piece_id=QUGILQI63G7CTO6FEKLQYPH3GTGYBFVIHFEGTQZ4FPNKVHIJ6O7A, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.637900207 (616h22m2.508983252s) [noise=true , address=77.90.31.11:4554 , piece_id=UN3ILRE7M3GJOWR6MZ5TKFIMFKJ64VHJI54ALMRVBBG5GM7ZVCGQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.639562833 (616h22m2.510645878s) [noise=true , address=145.79.227.50:4375 , piece_id=JYNH7FCK7S5J33MUIKIKBQL4LEXE2AUIGP27F7ILGDF6BCLYRVNA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.641683061 (616h22m2.512766116s) [noise=true , address=45.91.116.5:4297 , piece_id=2FPB4D776UYFTR4AL4NAWLKRC2NCAOECKIVAISSFV3MQ5EFIHFXA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.644611627 (616h22m2.515694672s) [noise=true , address=145.79.227.17:4970 , piece_id=IEZ3XCQ4ILSHKG3FMPUMRX3ED6K375LQGZDOIPO3RDXH3HPFEMGQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.646169363 (616h22m2.517252408s) [noise=true , address=145.79.227.45:4229 , piece_id=QNNEZTZLYLXB4CJHIQZ7XWFFNQTJHHGB4G6A6OWJPM5CULAT2MMA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.647613794 (616h22m2.518696849s) [noise=true , address=145.79.227.41:4085 , piece_id=RVNDIJAJBAPI2HUIH7INGP7743IZKZ6FJE4S46E5XVWMQF4XKXBA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.649442155 (616h22m2.52052521s) [noise=true , address=145.79.227.2:4113 , piece_id=4UE7ZTZQZ4W5HILOQWKBG5PZLOLKEPNTAVCQWB4HEIPHY3MBANBQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.650856880 (616h22m2.521939935s) [noise=true , address=145.79.227.12:4765 , piece_id=WD57BZBPHBVRZ4BJLY6JIRLQEESFXMAHZEHNKPQBIESPCZTAUBXQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.651427886 (616h22m2.522510921s) [noise=true , address=145.79.227.31:4798 , piece_id=UQJP3U5QFT5JHJ5DTP7QRZNVY2535FPVODKFOABGW3WP74NOPYDQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:04.306991452 (616h22m4.178074497s) [upload=540141]: Next returned nil segment
Dec 31 07:51:04.307028723 (616h22m4.178111768s) [upload=540141]: Waiting for error group managing segments...
Dec 31 07:51:04.637093559 (616h22m4.508176594s) [noise=true , address=145.79.227.5:4223 , piece_id=U7A2ZGU3CUSID72JOCL23CVC3BSYOYHRW6MTDHGFMQB2V34MY7XA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.639966499 (616h22m4.511049544s) [noise=true , address=145.79.227.2:4113 , piece_id=4UE7ZTZQZ4W5HILOQWKBG5PZLOLKEPNTAVCQWB4HEIPHY3MBANBQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.641266174 (616h22m4.51234922s) [noise=true , address=145.79.227.13:4804 , piece_id=QUGILQI63G7CTO6FEKLQYPH3GTGYBFVIHFEGTQZ4FPNKVHIJ6O7A, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.648310517 (616h22m4.519393562s) [noise=true , address=145.79.227.45:4229 , piece_id=QNNEZTZLYLXB4CJHIQZ7XWFFNQTJHHGB4G6A6OWJPM5CULAT2MMA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.656511744 (616h22m4.527594789s) [noise=true , address=145.79.227.41:4085 , piece_id=RVNDIJAJBAPI2HUIH7INGP7743IZKZ6FJE4S46E5XVWMQF4XKXBA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.663003926 (616h22m4.534086972s) [noise=true , address=145.79.227.36:4517 , piece_id=4GCERGAIUOSKTH533BBCFGSZUZLTOIJJWAMTFEKUMBZRQARYIK6Q, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.663531511 (616h22m4.534614556s) [noise=true , address=145.79.227.31:4798 , piece_id=UQJP3U5QFT5JHJ5DTP7QRZNVY2535FPVODKFOABGW3WP74NOPYDQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.669874068 (616h22m4.540957103s) [noise=true , address=145.79.227.12:4765 , piece_id=WD57BZBPHBVRZ4BJLY6JIRLQEESFXMAHZEHNKPQBIESPCZTAUBXQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.673966300 (616h22m4.545049335s) [noise=true , address=145.79.227.24:4582 , piece_id=QSMFJVHVRFVFDL4DT2UHLWMVF66HTQQ3X3HETXKVL4JERZFCP4QA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.728762481 (616h22m4.599845536s) [noise=true , address=145.79.227.50:4375 , piece_id=JYNH7FCK7S5J33MUIKIKBQL4LEXE2AUIGP27F7ILGDF6BCLYRVNA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.731376869 (616h22m4.602459904s) [noise=true , address=145.79.227.17:4970 , piece_id=IEZ3XCQ4ILSHKG3FMPUMRX3ED6K375LQGZDOIPO3RDXH3HPFEMGQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:05.026421641 (616h22m4.897504676s) [noise=true , address=45.91.116.5:4297 , piece_id=2FPB4D776UYFTR4AL4NAWLKRC2NCAOECKIVAISSFV3MQ5EFIHFXA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:05.071647599 (616h22m4.942730644s) [noise=true , address=77.90.31.11:4554 , piece_id=UN3ILRE7M3GJOWR6MZ5TKFIMFKJ64VHJI54ALMRVBBG5GM7ZVCGQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:05.071675381 (616h22m4.942758426s) [seg_pos={0 0}, upload=540141]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:51:05.072120689 (616h22m4.943203754s) [noise=true , address=145.79.227.29:4943 , piece_id=7ZCXKYQET76RXQ4HVQJFLZBUSXCVVYYQO7B7VH5EFQX5RBFLD4QA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF
Dec 31 07:51:05.072132842 (616h22m4.943215887s) [noise=true , address=145.79.227.29:4943 , piece_id=7ZCXKYQET76RXQ4HVQJFLZBUSXCVVYYQO7B7VH5EFQX5RBFLD4QA, seg_pos={0 0}, upload=540141]: Long tail cancellation due to optimal threshold
Dec 31 07:51:05.072587348 (616h22m4.943670383s) [noise=true , address=145.79.227.44:4193 , piece_id=AIE6JX7EQMVED7XDIWNQ65TZJTJVCY5CR7IWQCE37I3M6RYWVSKQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 31 07:51:05.072596675 (616h22m4.943679721s) [noise=true , address=145.79.227.44:4193 , piece_id=AIE6JX7EQMVED7XDIWNQ65TZJTJVCY5CR7IWQCE37I3M6RYWVSKQ, seg_pos={0 0}, upload=540141]: Long tail cancellation due to optimal threshold
Dec 31 07:51:05.072766098 (616h22m4.943849153s) [noise=true , address=77.90.31.16:4735 , piece_id=GPHX2PRAK34RGV4QSKDA2EP377FG26VY4RQAEYILTXVNXKYQ4IRA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 31 07:51:05.072775756 (616h22m4.943858801s) [noise=true , address=77.90.31.16:4735 , piece_id=GPHX2PRAK34RGV4QSKDA2EP377FG26VY4RQAEYILTXVNXKYQ4IRA, seg_pos={0 0}, upload=540141]: Long tail cancellation due to optimal threshold
Dec 31 07:51:05.072798109 (616h22m4.943881154s) [seg_pos={0 0}, upload=540141]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:51:05.072935842 (616h22m4.944018887s) [upload=540141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:51:05.072944167 (616h22m4.944027212s) [upload=540141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:51:05.366342304 (616h22m5.237425349s) [upload=540141]: Done uploading segments...
Dec 31 07:51:05.366368162 (616h22m5.237451207s) [upload=540141]: Upload finished. err: <nil>
Duration: 5.828862434s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
2.257563526s trace:
https://jaeger.dev.storj.io/trace/59c3b821b75eba3a
FAIL (2025-12-31 07:51:45 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 2.257588124s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...X................................X.........X............X] duration:
5.002128685s trace:
https://jaeger.dev.storj.io/trace/4dbc5f818d3c3b7
FAIL (2025-12-31 07:51:40 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002151337s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
6.512753934s trace:
https://jaeger.dev.storj.io/trace/5dd0609e3a2ac547
FAIL (2025-12-31 07:51:32 UTC)
orbiter warning: context deadline exceeded
Duration: 6.512780924s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [..X....X.X.X.X.X......XXX.X....XXX..X...........X......X.X.X] duration:
30.003135802s trace:
https://jaeger.dev.storj.io/trace/b9a66a0708d887e
FAIL (2025-12-31 07:51:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 07:51:02.386794824 (616h22m2.257877859s) [upload=540141]: Starting upload
Dec 31 07:51:02.386883673 (616h22m2.257966748s) [upload=540141]: Done starting upload
Dec 31 07:51:02.387061752 (616h22m2.258144807s) [upload=540141]: Uploading segments...
Dec 31 07:51:02.387169487 (616h22m2.258252532s) [upload=540141]: Got next segment. Inline: false
Dec 31 07:51:02.387235102 (616h22m2.258318157s) [upload=540141]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:51:02.387242135 (616h22m2.25832518s) [upload=540141]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:51:02.631477047 (616h22m2.502560102s) [seg_pos={0 0}, upload=540141]: Begin upload segment...
Dec 31 07:51:02.631536671 (616h22m2.502619716s) [seg_pos={0 0}, upload=540141]: Done begin upload segment.
Dec 31 07:51:02.631572469 (616h22m2.502655524s) [noise=true , address=145.79.227.5:4223 , piece_id=U7A2ZGU3CUSID72JOCL23CVC3BSYOYHRW6MTDHGFMQB2V34MY7XA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.633123201 (616h22m2.504206246s) [noise=true , address=145.79.227.44:4193 , piece_id=AIE6JX7EQMVED7XDIWNQ65TZJTJVCY5CR7IWQCE37I3M6RYWVSKQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.634110732 (616h22m2.505193777s) [noise=true , address=145.79.227.36:4517 , piece_id=4GCERGAIUOSKTH533BBCFGSZUZLTOIJJWAMTFEKUMBZRQARYIK6Q, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.634670077 (616h22m2.505753122s) [noise=true , address=145.79.227.29:4943 , piece_id=7ZCXKYQET76RXQ4HVQJFLZBUSXCVVYYQO7B7VH5EFQX5RBFLD4QA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.635257716 (616h22m2.506340761s) [noise=true , address=77.90.31.16:4735 , piece_id=GPHX2PRAK34RGV4QSKDA2EP377FG26VY4RQAEYILTXVNXKYQ4IRA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.636355436 (616h22m2.507438471s) [noise=true , address=145.79.227.24:4582 , piece_id=QSMFJVHVRFVFDL4DT2UHLWMVF66HTQQ3X3HETXKVL4JERZFCP4QA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.637031744 (616h22m2.508114789s) [noise=true , address=145.79.227.13:4804 , piece_id=QUGILQI63G7CTO6FEKLQYPH3GTGYBFVIHFEGTQZ4FPNKVHIJ6O7A, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.637900207 (616h22m2.508983252s) [noise=true , address=77.90.31.11:4554 , piece_id=UN3ILRE7M3GJOWR6MZ5TKFIMFKJ64VHJI54ALMRVBBG5GM7ZVCGQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.639562833 (616h22m2.510645878s) [noise=true , address=145.79.227.50:4375 , piece_id=JYNH7FCK7S5J33MUIKIKBQL4LEXE2AUIGP27F7ILGDF6BCLYRVNA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.641683061 (616h22m2.512766116s) [noise=true , address=45.91.116.5:4297 , piece_id=2FPB4D776UYFTR4AL4NAWLKRC2NCAOECKIVAISSFV3MQ5EFIHFXA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.644611627 (616h22m2.515694672s) [noise=true , address=145.79.227.17:4970 , piece_id=IEZ3XCQ4ILSHKG3FMPUMRX3ED6K375LQGZDOIPO3RDXH3HPFEMGQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.646169363 (616h22m2.517252408s) [noise=true , address=145.79.227.45:4229 , piece_id=QNNEZTZLYLXB4CJHIQZ7XWFFNQTJHHGB4G6A6OWJPM5CULAT2MMA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.647613794 (616h22m2.518696849s) [noise=true , address=145.79.227.41:4085 , piece_id=RVNDIJAJBAPI2HUIH7INGP7743IZKZ6FJE4S46E5XVWMQF4XKXBA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.649442155 (616h22m2.52052521s) [noise=true , address=145.79.227.2:4113 , piece_id=4UE7ZTZQZ4W5HILOQWKBG5PZLOLKEPNTAVCQWB4HEIPHY3MBANBQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.650856880 (616h22m2.521939935s) [noise=true , address=145.79.227.12:4765 , piece_id=WD57BZBPHBVRZ4BJLY6JIRLQEESFXMAHZEHNKPQBIESPCZTAUBXQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.651427886 (616h22m2.522510921s) [noise=true , address=145.79.227.31:4798 , piece_id=UQJP3U5QFT5JHJ5DTP7QRZNVY2535FPVODKFOABGW3WP74NOPYDQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:04.306991452 (616h22m4.178074497s) [upload=540141]: Next returned nil segment
Dec 31 07:51:04.307028723 (616h22m4.178111768s) [upload=540141]: Waiting for error group managing segments...
Dec 31 07:51:04.637093559 (616h22m4.508176594s) [noise=true , address=145.79.227.5:4223 , piece_id=U7A2ZGU3CUSID72JOCL23CVC3BSYOYHRW6MTDHGFMQB2V34MY7XA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.639966499 (616h22m4.511049544s) [noise=true , address=145.79.227.2:4113 , piece_id=4UE7ZTZQZ4W5HILOQWKBG5PZLOLKEPNTAVCQWB4HEIPHY3MBANBQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.641266174 (616h22m4.51234922s) [noise=true , address=145.79.227.13:4804 , piece_id=QUGILQI63G7CTO6FEKLQYPH3GTGYBFVIHFEGTQZ4FPNKVHIJ6O7A, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.648310517 (616h22m4.519393562s) [noise=true , address=145.79.227.45:4229 , piece_id=QNNEZTZLYLXB4CJHIQZ7XWFFNQTJHHGB4G6A6OWJPM5CULAT2MMA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.656511744 (616h22m4.527594789s) [noise=true , address=145.79.227.41:4085 , piece_id=RVNDIJAJBAPI2HUIH7INGP7743IZKZ6FJE4S46E5XVWMQF4XKXBA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.663003926 (616h22m4.534086972s) [noise=true , address=145.79.227.36:4517 , piece_id=4GCERGAIUOSKTH533BBCFGSZUZLTOIJJWAMTFEKUMBZRQARYIK6Q, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.663531511 (616h22m4.534614556s) [noise=true , address=145.79.227.31:4798 , piece_id=UQJP3U5QFT5JHJ5DTP7QRZNVY2535FPVODKFOABGW3WP74NOPYDQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.669874068 (616h22m4.540957103s) [noise=true , address=145.79.227.12:4765 , piece_id=WD57BZBPHBVRZ4BJLY6JIRLQEESFXMAHZEHNKPQBIESPCZTAUBXQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.673966300 (616h22m4.545049335s) [noise=true , address=145.79.227.24:4582 , piece_id=QSMFJVHVRFVFDL4DT2UHLWMVF66HTQQ3X3HETXKVL4JERZFCP4QA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.728762481 (616h22m4.599845536s) [noise=true , address=145.79.227.50:4375 , piece_id=JYNH7FCK7S5J33MUIKIKBQL4LEXE2AUIGP27F7ILGDF6BCLYRVNA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.731376869 (616h22m4.602459904s) [noise=true , address=145.79.227.17:4970 , piece_id=IEZ3XCQ4ILSHKG3FMPUMRX3ED6K375LQGZDOIPO3RDXH3HPFEMGQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:05.026421641 (616h22m4.897504676s) [noise=true , address=45.91.116.5:4297 , piece_id=2FPB4D776UYFTR4AL4NAWLKRC2NCAOECKIVAISSFV3MQ5EFIHFXA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:05.071647599 (616h22m4.942730644s) [noise=true , address=77.90.31.11:4554 , piece_id=UN3ILRE7M3GJOWR6MZ5TKFIMFKJ64VHJI54ALMRVBBG5GM7ZVCGQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:05.071675381 (616h22m4.942758426s) [seg_pos={0 0}, upload=540141]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:51:05.072120689 (616h22m4.943203754s) [noise=true , address=145.79.227.29:4943 , piece_id=7ZCXKYQET76RXQ4HVQJFLZBUSXCVVYYQO7B7VH5EFQX5RBFLD4QA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF
Dec 31 07:51:05.072132842 (616h22m4.943215887s) [noise=true , address=145.79.227.29:4943 , piece_id=7ZCXKYQET76RXQ4HVQJFLZBUSXCVVYYQO7B7VH5EFQX5RBFLD4QA, seg_pos={0 0}, upload=540141]: Long tail cancellation due to optimal threshold
Dec 31 07:51:05.072587348 (616h22m4.943670383s) [noise=true , address=145.79.227.44:4193 , piece_id=AIE6JX7EQMVED7XDIWNQ65TZJTJVCY5CR7IWQCE37I3M6RYWVSKQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 31 07:51:05.072596675 (616h22m4.943679721s) [noise=true , address=145.79.227.44:4193 , piece_id=AIE6JX7EQMVED7XDIWNQ65TZJTJVCY5CR7IWQCE37I3M6RYWVSKQ, seg_pos={0 0}, upload=540141]: Long tail cancellation due to optimal threshold
Dec 31 07:51:05.072766098 (616h22m4.943849153s) [noise=true , address=77.90.31.16:4735 , piece_id=GPHX2PRAK34RGV4QSKDA2EP377FG26VY4RQAEYILTXVNXKYQ4IRA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 31 07:51:05.072775756 (616h22m4.943858801s) [noise=true , address=77.90.31.16:4735 , piece_id=GPHX2PRAK34RGV4QSKDA2EP377FG26VY4RQAEYILTXVNXKYQ4IRA, seg_pos={0 0}, upload=540141]: Long tail cancellation due to optimal threshold
Dec 31 07:51:05.072798109 (616h22m4.943881154s) [seg_pos={0 0}, upload=540141]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:51:05.072935842 (616h22m4.944018887s) [upload=540141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:51:05.072944167 (616h22m4.944027212s) [upload=540141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:51:05.366342304 (616h22m5.237425349s) [upload=540141]: Done uploading segments...
Dec 31 07:51:05.366368162 (616h22m5.237451207s) [upload=540141]: Upload finished. err: <nil>
Duration: 30.003175628s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........X............X...................................] duration:
24.113122358s trace:
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:51:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505 , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147 , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965 , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656 , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500 , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673 , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977 , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087 , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620 , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896 , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370 , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228 , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067 , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784 , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709 , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707 , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186 , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 24.113148528s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
6.236366316s trace:
https://jaeger.dev.storj.io/trace/5dd0609e3a2ac547
FAIL (2025-12-31 07:51:13 UTC)
orbiter warning: context deadline exceeded
Duration: 6.236392166s
edge
gateway
us-select-1
|