Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [..X.X.X.....X......XX.X..XXX.X.XXX.XX.XX...X.X........X.X.X.] duration:
6.052888016s trace:
https://jaeger.dev.storj.io/trace/354f5c93194630fe
FAIL (2025-12-28 15:45:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 6JZJDZI7FXSZW3H2R25DJYLS5GIZM2XJJIO6FCNN6Q6Y63V7TZOQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: KMQ7XW76YAJA5HDEKKFK22GHKMAZHECR6DKBRVDEBUJE63F6MMOQ) context deadline exceeded
======LOGS======
segments...
Dec 28 15:44:23.228864797 (552h15m23.099947852s) [upload=484187]: Got next segment. Inline: false
Dec 28 15:44:23.228965468 (552h15m23.100048503s) [upload=484187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:44:23.228972282 (552h15m23.100055317s) [upload=484187]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:44:23.473522066 (552h15m23.344605101s) [seg_pos={0 0}, upload=484187]: Begin upload segment...
Dec 28 15:44:23.473581239 (552h15m23.344664274s) [seg_pos={0 0}, upload=484187]: Done begin upload segment.
Dec 28 15:44:23.473646121 (552h15m23.344729156s) [noise=true , address=145.79.227.32:4834 , piece_id=XWAALBLYIILFWOWOMHB3FM6LNKQLNSDTQQ7IGWJLI73J7S4T2BPA, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.473663134 (552h15m23.344746169s) [noise=true , address=45.91.116.6:4974 , piece_id=GQN5ZGZBHTVMGJIF25WZVAUGRYVDVESYCPRAORIVMTKFELUD4NYQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.474181331 (552h15m23.345264376s) [noise=true , address=45.91.116.5:4300 , piece_id=6VVNTJLAH6KTCNUCJBUJMDVWZXFJQJQ3EXXCNSDSDCESSPQBVFOQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.474675652 (552h15m23.345758687s) [noise=true , address=145.79.227.38:4979 , piece_id=ICQUTWM7KWDEGV6MQVLRNMHL4B55EEBI6IILHUZ32DQIKLIIRZEA, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.475073770 (552h15m23.346156805s) [noise=true , address=145.79.227.18:4007 , piece_id=YVNQUVKS4NHHSQYUDNHUNVLNFQITS56QLX77DFDGXBGT2XNKI7KQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.475456669 (552h15m23.346539734s) [noise=true , address=145.79.227.45:4231 , piece_id=PTWRS7JWHN4NKEU7JBHTPU272X6L7KFX7Z5IVQJ6FZSHVQGAJZOQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.475465355 (552h15m23.3465484s) [noise=true , address=77.90.31.10:4519 , piece_id=K2R5W7KM64SQCMS7RHKFBSBKJDUOCTSVJ6PRIX5BRCVMPOI55M4Q, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.475866389 (552h15m23.346949434s) [noise=true , address=145.79.227.48:4302 , piece_id=Y7ZIYCSKVNRTQYDS2JAVEBPFIDUGUVUPSBUJ5VDAH7FOR6QOC54A, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.476195155 (552h15m23.34727819s) [noise=true , address=145.79.227.52:4445 , piece_id=DWK5YFNHIHKITVDLWI3VZVAAAHNX67RZHVIVYLU2FEC3IZB3QXGA, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.476489416 (552h15m23.347572461s) [noise=true , address=145.79.227.51:4409 , piece_id=V4BNGIDOFEZ2PS6HHMAZFQSMUXZT6DVLHE6U3CMFPBPGR2DMLZGQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.476768978 (552h15m23.347852013s) [noise=true , address=45.91.116.12:4228 , piece_id=6JZJDZI7FXSZW3H2R25DJYLS5GIZM2XJJIO6FCNN6Q6Y63V7TZOQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.477082856 (552h15m23.348165891s) [noise=true , address=145.79.227.47:4266 , piece_id=KMT6CODJY2TIU4LRVVH7EBNR5W43IJPTSQHQNSTCM665KY4BON4A, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.477730388 (552h15m23.348813433s) [noise=true , address=45.91.116.10:4083 , piece_id=KMQ7XW76YAJA5HDEKKFK22GHKMAZHECR6DKBRVDEBUJE63F6MMOQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.479783297 (552h15m23.350866342s) [noise=true , address=145.79.227.11:4731 , piece_id=YT3FOQNK2UL5M57HEZ6ESBTVMST2QS6WCRDLQ3MRVLMWPVWGWGWQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.482206953 (552h15m23.353289988s) [noise=true , address=45.91.116.7:4009 , piece_id=5YUBW6YCUMNKCGJGDZNIX4SE7D2MCIXPKNGZ7PDCP72U3X6PT33A, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.484736129 (552h15m23.355819164s) [noise=true , address=145.79.227.37:4567 , piece_id=TNL66LEKN2EAVIZXCJFML7BIEL2AR7OVD43Y3LDIEFBGVEPLVR7Q, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:25.195219194 (552h15m25.06630223s) [upload=484187]: Next returned nil segment
Dec 28 15:44:25.195233162 (552h15m25.066316197s) [upload=484187]: Waiting for error group managing segments...
Dec 28 15:44:25.479305062 (552h15m25.350388097s) [noise=true , address=145.79.227.18:4007 , piece_id=YVNQUVKS4NHHSQYUDNHUNVLNFQITS56QLX77DFDGXBGT2XNKI7KQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.479557773 (552h15m25.350640808s) [noise=true , address=145.79.227.38:4979 , piece_id=ICQUTWM7KWDEGV6MQVLRNMHL4B55EEBI6IILHUZ32DQIKLIIRZEA, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.495908758 (552h15m25.366991794s) [noise=true , address=145.79.227.11:4731 , piece_id=YT3FOQNK2UL5M57HEZ6ESBTVMST2QS6WCRDLQ3MRVLMWPVWGWGWQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.498335489 (552h15m25.369418524s) [noise=true , address=145.79.227.37:4567 , piece_id=TNL66LEKN2EAVIZXCJFML7BIEL2AR7OVD43Y3LDIEFBGVEPLVR7Q, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.498557702 (552h15m25.369640748s) [noise=true , address=145.79.227.47:4266 , piece_id=KMT6CODJY2TIU4LRVVH7EBNR5W43IJPTSQHQNSTCM665KY4BON4A, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.502042959 (552h15m25.373125994s) [noise=true , address=145.79.227.45:4231 , piece_id=PTWRS7JWHN4NKEU7JBHTPU272X6L7KFX7Z5IVQJ6FZSHVQGAJZOQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.507488137 (552h15m25.378571172s) [noise=true , address=145.79.227.51:4409 , piece_id=V4BNGIDOFEZ2PS6HHMAZFQSMUXZT6DVLHE6U3CMFPBPGR2DMLZGQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.512839918 (552h15m25.383922963s) [noise=true , address=145.79.227.52:4445 , piece_id=DWK5YFNHIHKITVDLWI3VZVAAAHNX67RZHVIVYLU2FEC3IZB3QXGA, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.514591643 (552h15m25.385674678s) [noise=true , address=145.79.227.48:4302 , piece_id=Y7ZIYCSKVNRTQYDS2JAVEBPFIDUGUVUPSBUJ5VDAH7FOR6QOC54A, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.528475671 (552h15m25.399558716s) [noise=true , address=145.79.227.32:4834 , piece_id=XWAALBLYIILFWOWOMHB3FM6LNKQLNSDTQQ7IGWJLI73J7S4T2BPA, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.949610562 (552h15m25.820693617s) [noise=true , address=45.91.116.12:4228 , piece_id=6JZJDZI7FXSZW3H2R25DJYLS5GIZM2XJJIO6FCNN6Q6Y63V7TZOQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:26.470746833 (552h15m26.341829878s) [noise=true , address=45.91.116.7:4009 , piece_id=5YUBW6YCUMNKCGJGDZNIX4SE7D2MCIXPKNGZ7PDCP72U3X6PT33A, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:26.557979107 (552h15m26.429062142s) [noise=true , address=45.91.116.10:4083 , piece_id=KMQ7XW76YAJA5HDEKKFK22GHKMAZHECR6DKBRVDEBUJE63F6MMOQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:26.557998314 (552h15m26.429081349s) [seg_pos={0 0}, upload=484187]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:44:26.558185981 (552h15m26.429269016s) [noise=true , address=45.91.116.5:4300 , piece_id=6VVNTJLAH6KTCNUCJBUJMDVWZXFJQJQ3EXXCNSDSDCESSPQBVFOQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 28 15:44:26.558193195 (552h15m26.429276241s) [noise=true , address=45.91.116.5:4300 , piece_id=6VVNTJLAH6KTCNUCJBUJMDVWZXFJQJQ3EXXCNSDSDCESSPQBVFOQ, seg_pos={0 0}, upload=484187]: Long tail cancellation due to optimal threshold
Dec 28 15:44:26.558306611 (552h15m26.429389656s) [noise=true , address=77.90.31.10:4519 , piece_id=K2R5W7KM64SQCMS7RHKFBSBKJDUOCTSVJ6PRIX5BRCVMPOI55M4Q, seg_pos={0 0}, upload=484187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:42294->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:42294->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42294->77.90.31.10:4519: use of closed network connection
Dec 28 15:44:26.558312673 (552h15m26.429395708s) [noise=true , address=77.90.31.10:4519 , piece_id=K2R5W7KM64SQCMS7RHKFBSBKJDUOCTSVJ6PRIX5BRCVMPOI55M4Q, seg_pos={0 0}, upload=484187]: Long tail cancellation due to optimal threshold
Dec 28 15:44:26.558556727 (552h15m26.429639782s) [noise=true , address=45.91.116.6:4974 , piece_id=GQN5ZGZBHTVMGJIF25WZVAUGRYVDVESYCPRAORIVMTKFELUD4NYQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52484->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52484->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52484->45.91.116.6:4974: use of closed network connection
Dec 28 15:44:26.558577006 (552h15m26.429660061s) [noise=true , address=45.91.116.6:4974 , piece_id=GQN5ZGZBHTVMGJIF25WZVAUGRYVDVESYCPRAORIVMTKFELUD4NYQ, seg_pos={0 0}, upload=484187]: Long tail cancellation due to optimal threshold
Dec 28 15:44:26.558613987 (552h15m26.429697022s) [seg_pos={0 0}, upload=484187]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:44:26.558755165 (552h15m26.4298382s) [upload=484187]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:44:26.558762529 (552h15m26.429845564s) [upload=484187]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:44:26.851863402 (552h15m26.722946437s) [upload=484187]: Done uploading segments...
Dec 28 15:44:26.851883069 (552h15m26.722966114s) [upload=484187]: Upload finished. err: <nil>
Duration: 6.052947028s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
14.822894156s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 15:45:28 UTC)
orbiter warning: context deadline exceeded
Duration: 14.8229222s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............X................................X............] duration:
15.229963572s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:45:23 UTC)
orbiter warning: context deadline exceeded
Duration: 15.229988419s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....................X....X...X........................X....] duration:
42.668209424s trace:
https://jaeger.dev.storj.io/trace/3cc795a000779b86
FAIL (2025-12-28 15:45:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ) context deadline exceeded; piecestore: (Node ID: 128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, Piece ID: GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q) context deadline exceeded; piecestore: (Node ID: 12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, Piece ID: TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A) context deadline exceeded
======LOGS======
48.156469147 (552h11m48.027552193s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157156416 (552h11m48.028239461s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157556748 (552h11m48.028639793s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157945468 (552h11m48.029028503s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158645151 (552h11m48.029728196s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158842136 (552h11m48.029925171s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158983275 (552h11m48.03006631s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159344152 (552h11m48.030427187s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159350725 (552h11m48.03043376s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159710760 (552h11m48.030793795s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159732391 (552h11m48.030815437s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160068310 (552h11m48.031151346s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160439438 (552h11m48.031522473s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160813179 (552h11m48.031896225s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160839249 (552h11m48.031922284s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161271813 (552h11m48.032354848s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161765813 (552h11m48.032848848s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:49.456221594 (552h11m49.327304629s) [upload=484135]: Next returned nil segment
Dec 28 15:40:49.456235460 (552h11m49.327318495s) [upload=484135]: Waiting for error group managing segments...
Dec 28 15:40:49.789381987 (552h11m49.660465032s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.818159755 (552h11m49.6892428s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.901730607 (552h11m49.772813642s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.297735267 (552h11m50.168818312s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.866598958 (552h11m50.737682003s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.955467169 (552h11m50.826550214s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.029562809 (552h11m50.900645854s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.033202229 (552h11m50.904285264s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.119305864 (552h11m50.990388909s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.271054029 (552h11m51.142137074s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.374827580 (552h11m51.245910626s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.380607978 (552h11m51.251691013s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.500440018 (552h11m51.371523063s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688250240 (552h11m51.559333285s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688275438 (552h11m51.559358473s) [seg_pos={0 0}, upload=484135]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:40:51.688501768 (552h11m51.559584804s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688512109 (552h11m51.559595144s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688551694 (552h11m51.559634729s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688560741 (552h11m51.559643786s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688768767 (552h11m51.559851802s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWqAGD7Vrj2yzEVyNbfPp278Gga42xPLyAcdzGvUSuz3EgJYh): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688776031 (552h11m51.559859066s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688811729 (552h11m51.559894784s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688820707 (552h11m51.559903752s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688840654 (552h11m51.559923689s) [seg_pos={0 0}, upload=484135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:40:51.688958800 (552h11m51.560041845s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:40:51.688965582 (552h11m51.560048617s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:40:51.990040599 (552h11m51.861123664s) [upload=484135]: Done uploading segments...
Dec 28 15:40:51.990091757 (552h11m51.861174802s) [upload=484135]: Upload finished. err: <nil>
Duration: 42.668230714s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..XX...XXX] duration:
5.00134606s trace:
https://jaeger.dev.storj.io/trace/64d166e63eaa7bab
FAIL (2025-12-28 15:44:59 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001369604s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [...X.X.X.....X......XX.X..XXX.X.XXX.XX.XX...X.X........X.X.X] duration:
30.004158481s trace:
https://jaeger.dev.storj.io/trace/354f5c93194630fe
FAIL (2025-12-28 15:44:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 6JZJDZI7FXSZW3H2R25DJYLS5GIZM2XJJIO6FCNN6Q6Y63V7TZOQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: KMQ7XW76YAJA5HDEKKFK22GHKMAZHECR6DKBRVDEBUJE63F6MMOQ) context deadline exceeded
======LOGS======
segments...
Dec 28 15:44:23.228864797 (552h15m23.099947852s) [upload=484187]: Got next segment. Inline: false
Dec 28 15:44:23.228965468 (552h15m23.100048503s) [upload=484187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:44:23.228972282 (552h15m23.100055317s) [upload=484187]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:44:23.473522066 (552h15m23.344605101s) [seg_pos={0 0}, upload=484187]: Begin upload segment...
Dec 28 15:44:23.473581239 (552h15m23.344664274s) [seg_pos={0 0}, upload=484187]: Done begin upload segment.
Dec 28 15:44:23.473646121 (552h15m23.344729156s) [noise=true , address=145.79.227.32:4834 , piece_id=XWAALBLYIILFWOWOMHB3FM6LNKQLNSDTQQ7IGWJLI73J7S4T2BPA, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.473663134 (552h15m23.344746169s) [noise=true , address=45.91.116.6:4974 , piece_id=GQN5ZGZBHTVMGJIF25WZVAUGRYVDVESYCPRAORIVMTKFELUD4NYQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.474181331 (552h15m23.345264376s) [noise=true , address=45.91.116.5:4300 , piece_id=6VVNTJLAH6KTCNUCJBUJMDVWZXFJQJQ3EXXCNSDSDCESSPQBVFOQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.474675652 (552h15m23.345758687s) [noise=true , address=145.79.227.38:4979 , piece_id=ICQUTWM7KWDEGV6MQVLRNMHL4B55EEBI6IILHUZ32DQIKLIIRZEA, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.475073770 (552h15m23.346156805s) [noise=true , address=145.79.227.18:4007 , piece_id=YVNQUVKS4NHHSQYUDNHUNVLNFQITS56QLX77DFDGXBGT2XNKI7KQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.475456669 (552h15m23.346539734s) [noise=true , address=145.79.227.45:4231 , piece_id=PTWRS7JWHN4NKEU7JBHTPU272X6L7KFX7Z5IVQJ6FZSHVQGAJZOQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.475465355 (552h15m23.3465484s) [noise=true , address=77.90.31.10:4519 , piece_id=K2R5W7KM64SQCMS7RHKFBSBKJDUOCTSVJ6PRIX5BRCVMPOI55M4Q, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.475866389 (552h15m23.346949434s) [noise=true , address=145.79.227.48:4302 , piece_id=Y7ZIYCSKVNRTQYDS2JAVEBPFIDUGUVUPSBUJ5VDAH7FOR6QOC54A, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.476195155 (552h15m23.34727819s) [noise=true , address=145.79.227.52:4445 , piece_id=DWK5YFNHIHKITVDLWI3VZVAAAHNX67RZHVIVYLU2FEC3IZB3QXGA, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.476489416 (552h15m23.347572461s) [noise=true , address=145.79.227.51:4409 , piece_id=V4BNGIDOFEZ2PS6HHMAZFQSMUXZT6DVLHE6U3CMFPBPGR2DMLZGQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.476768978 (552h15m23.347852013s) [noise=true , address=45.91.116.12:4228 , piece_id=6JZJDZI7FXSZW3H2R25DJYLS5GIZM2XJJIO6FCNN6Q6Y63V7TZOQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.477082856 (552h15m23.348165891s) [noise=true , address=145.79.227.47:4266 , piece_id=KMT6CODJY2TIU4LRVVH7EBNR5W43IJPTSQHQNSTCM665KY4BON4A, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.477730388 (552h15m23.348813433s) [noise=true , address=45.91.116.10:4083 , piece_id=KMQ7XW76YAJA5HDEKKFK22GHKMAZHECR6DKBRVDEBUJE63F6MMOQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.479783297 (552h15m23.350866342s) [noise=true , address=145.79.227.11:4731 , piece_id=YT3FOQNK2UL5M57HEZ6ESBTVMST2QS6WCRDLQ3MRVLMWPVWGWGWQ, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.482206953 (552h15m23.353289988s) [noise=true , address=45.91.116.7:4009 , piece_id=5YUBW6YCUMNKCGJGDZNIX4SE7D2MCIXPKNGZ7PDCP72U3X6PT33A, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:23.484736129 (552h15m23.355819164s) [noise=true , address=145.79.227.37:4567 , piece_id=TNL66LEKN2EAVIZXCJFML7BIEL2AR7OVD43Y3LDIEFBGVEPLVR7Q, seg_pos={0 0}, upload=484187]: Uploading piece...
Dec 28 15:44:25.195219194 (552h15m25.06630223s) [upload=484187]: Next returned nil segment
Dec 28 15:44:25.195233162 (552h15m25.066316197s) [upload=484187]: Waiting for error group managing segments...
Dec 28 15:44:25.479305062 (552h15m25.350388097s) [noise=true , address=145.79.227.18:4007 , piece_id=YVNQUVKS4NHHSQYUDNHUNVLNFQITS56QLX77DFDGXBGT2XNKI7KQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.479557773 (552h15m25.350640808s) [noise=true , address=145.79.227.38:4979 , piece_id=ICQUTWM7KWDEGV6MQVLRNMHL4B55EEBI6IILHUZ32DQIKLIIRZEA, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.495908758 (552h15m25.366991794s) [noise=true , address=145.79.227.11:4731 , piece_id=YT3FOQNK2UL5M57HEZ6ESBTVMST2QS6WCRDLQ3MRVLMWPVWGWGWQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.498335489 (552h15m25.369418524s) [noise=true , address=145.79.227.37:4567 , piece_id=TNL66LEKN2EAVIZXCJFML7BIEL2AR7OVD43Y3LDIEFBGVEPLVR7Q, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.498557702 (552h15m25.369640748s) [noise=true , address=145.79.227.47:4266 , piece_id=KMT6CODJY2TIU4LRVVH7EBNR5W43IJPTSQHQNSTCM665KY4BON4A, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.502042959 (552h15m25.373125994s) [noise=true , address=145.79.227.45:4231 , piece_id=PTWRS7JWHN4NKEU7JBHTPU272X6L7KFX7Z5IVQJ6FZSHVQGAJZOQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.507488137 (552h15m25.378571172s) [noise=true , address=145.79.227.51:4409 , piece_id=V4BNGIDOFEZ2PS6HHMAZFQSMUXZT6DVLHE6U3CMFPBPGR2DMLZGQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.512839918 (552h15m25.383922963s) [noise=true , address=145.79.227.52:4445 , piece_id=DWK5YFNHIHKITVDLWI3VZVAAAHNX67RZHVIVYLU2FEC3IZB3QXGA, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.514591643 (552h15m25.385674678s) [noise=true , address=145.79.227.48:4302 , piece_id=Y7ZIYCSKVNRTQYDS2JAVEBPFIDUGUVUPSBUJ5VDAH7FOR6QOC54A, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.528475671 (552h15m25.399558716s) [noise=true , address=145.79.227.32:4834 , piece_id=XWAALBLYIILFWOWOMHB3FM6LNKQLNSDTQQ7IGWJLI73J7S4T2BPA, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:25.949610562 (552h15m25.820693617s) [noise=true , address=45.91.116.12:4228 , piece_id=6JZJDZI7FXSZW3H2R25DJYLS5GIZM2XJJIO6FCNN6Q6Y63V7TZOQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:26.470746833 (552h15m26.341829878s) [noise=true , address=45.91.116.7:4009 , piece_id=5YUBW6YCUMNKCGJGDZNIX4SE7D2MCIXPKNGZ7PDCP72U3X6PT33A, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:26.557979107 (552h15m26.429062142s) [noise=true , address=45.91.116.10:4083 , piece_id=KMQ7XW76YAJA5HDEKKFK22GHKMAZHECR6DKBRVDEBUJE63F6MMOQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: <nil>
Dec 28 15:44:26.557998314 (552h15m26.429081349s) [seg_pos={0 0}, upload=484187]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:44:26.558185981 (552h15m26.429269016s) [noise=true , address=45.91.116.5:4300 , piece_id=6VVNTJLAH6KTCNUCJBUJMDVWZXFJQJQ3EXXCNSDSDCESSPQBVFOQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 28 15:44:26.558193195 (552h15m26.429276241s) [noise=true , address=45.91.116.5:4300 , piece_id=6VVNTJLAH6KTCNUCJBUJMDVWZXFJQJQ3EXXCNSDSDCESSPQBVFOQ, seg_pos={0 0}, upload=484187]: Long tail cancellation due to optimal threshold
Dec 28 15:44:26.558306611 (552h15m26.429389656s) [noise=true , address=77.90.31.10:4519 , piece_id=K2R5W7KM64SQCMS7RHKFBSBKJDUOCTSVJ6PRIX5BRCVMPOI55M4Q, seg_pos={0 0}, upload=484187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:42294->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:42294->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42294->77.90.31.10:4519: use of closed network connection
Dec 28 15:44:26.558312673 (552h15m26.429395708s) [noise=true , address=77.90.31.10:4519 , piece_id=K2R5W7KM64SQCMS7RHKFBSBKJDUOCTSVJ6PRIX5BRCVMPOI55M4Q, seg_pos={0 0}, upload=484187]: Long tail cancellation due to optimal threshold
Dec 28 15:44:26.558556727 (552h15m26.429639782s) [noise=true , address=45.91.116.6:4974 , piece_id=GQN5ZGZBHTVMGJIF25WZVAUGRYVDVESYCPRAORIVMTKFELUD4NYQ, seg_pos={0 0}, upload=484187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52484->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52484->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52484->45.91.116.6:4974: use of closed network connection
Dec 28 15:44:26.558577006 (552h15m26.429660061s) [noise=true , address=45.91.116.6:4974 , piece_id=GQN5ZGZBHTVMGJIF25WZVAUGRYVDVESYCPRAORIVMTKFELUD4NYQ, seg_pos={0 0}, upload=484187]: Long tail cancellation due to optimal threshold
Dec 28 15:44:26.558613987 (552h15m26.429697022s) [seg_pos={0 0}, upload=484187]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:44:26.558755165 (552h15m26.4298382s) [upload=484187]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:44:26.558762529 (552h15m26.429845564s) [upload=484187]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:44:26.851863402 (552h15m26.722946437s) [upload=484187]: Done uploading segments...
Dec 28 15:44:26.851883069 (552h15m26.722966114s) [upload=484187]: Upload finished. err: <nil>
Duration: 30.004181274s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
14.585963117s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 15:44:50 UTC)
orbiter warning: context deadline exceeded
Duration: 14.586063207s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............X................................X...........] duration:
12.907611804s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:44:47 UTC)
orbiter warning: context deadline exceeded
Duration: 12.907639196s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................X................................X..........] duration:
11.759311738s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:44:02 UTC)
orbiter warning: context deadline exceeded
Duration: 11.759334633s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [......................X....X...X........................X...] duration:
39.284379038s trace:
https://jaeger.dev.storj.io/trace/3cc795a000779b86
FAIL (2025-12-28 15:43:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ) context deadline exceeded; piecestore: (Node ID: 128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, Piece ID: GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q) context deadline exceeded; piecestore: (Node ID: 12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, Piece ID: TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A) context deadline exceeded
======LOGS======
48.156469147 (552h11m48.027552193s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157156416 (552h11m48.028239461s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157556748 (552h11m48.028639793s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157945468 (552h11m48.029028503s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158645151 (552h11m48.029728196s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158842136 (552h11m48.029925171s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158983275 (552h11m48.03006631s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159344152 (552h11m48.030427187s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159350725 (552h11m48.03043376s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159710760 (552h11m48.030793795s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159732391 (552h11m48.030815437s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160068310 (552h11m48.031151346s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160439438 (552h11m48.031522473s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160813179 (552h11m48.031896225s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160839249 (552h11m48.031922284s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161271813 (552h11m48.032354848s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161765813 (552h11m48.032848848s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:49.456221594 (552h11m49.327304629s) [upload=484135]: Next returned nil segment
Dec 28 15:40:49.456235460 (552h11m49.327318495s) [upload=484135]: Waiting for error group managing segments...
Dec 28 15:40:49.789381987 (552h11m49.660465032s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.818159755 (552h11m49.6892428s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.901730607 (552h11m49.772813642s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.297735267 (552h11m50.168818312s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.866598958 (552h11m50.737682003s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.955467169 (552h11m50.826550214s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.029562809 (552h11m50.900645854s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.033202229 (552h11m50.904285264s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.119305864 (552h11m50.990388909s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.271054029 (552h11m51.142137074s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.374827580 (552h11m51.245910626s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.380607978 (552h11m51.251691013s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.500440018 (552h11m51.371523063s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688250240 (552h11m51.559333285s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688275438 (552h11m51.559358473s) [seg_pos={0 0}, upload=484135]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:40:51.688501768 (552h11m51.559584804s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688512109 (552h11m51.559595144s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688551694 (552h11m51.559634729s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688560741 (552h11m51.559643786s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688768767 (552h11m51.559851802s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWqAGD7Vrj2yzEVyNbfPp278Gga42xPLyAcdzGvUSuz3EgJYh): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688776031 (552h11m51.559859066s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688811729 (552h11m51.559894784s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688820707 (552h11m51.559903752s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688840654 (552h11m51.559923689s) [seg_pos={0 0}, upload=484135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:40:51.688958800 (552h11m51.560041845s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:40:51.688965582 (552h11m51.560048617s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:40:51.990040599 (552h11m51.861123664s) [upload=484135]: Done uploading segments...
Dec 28 15:40:51.990091757 (552h11m51.861174802s) [upload=484135]: Upload finished. err: <nil>
Duration: 39.284396141s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [....X.X.X.....X......XX.X..XXX.X.XXX.XX.XX...X.X........X.X.] duration:
6.241403186s trace:
https://jaeger.dev.storj.io/trace/3862ec468111a894
FAIL (2025-12-28 15:43:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: PDPMJPLQESWKZSEVSFKELJZ57JWIT3WPB7FVMLWVBAAZHJTTIK4Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: COKORQ2MVWK5Y5HJVBYOGXPYLK5OLKPKVIK46LH2FDLHIUULHX4A) context deadline exceeded
======LOGS======
Dec 28 15:42:51.947870419 (552h13m51.818953454s) [upload=484167]: Starting upload
Dec 28 15:42:51.947931585 (552h13m51.81901464s) [upload=484167]: Done starting upload
Dec 28 15:42:51.948034371 (552h13m51.819117416s) [upload=484167]: Uploading segments...
Dec 28 15:42:51.948157325 (552h13m51.81924036s) [upload=484167]: Got next segment. Inline: false
Dec 28 15:42:51.948214774 (552h13m51.81929781s) [upload=484167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:42:51.948221377 (552h13m51.819304412s) [upload=484167]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:42:52.196339967 (552h13m52.067423002s) [seg_pos={0 0}, upload=484167]: Begin upload segment...
Dec 28 15:42:52.196385092 (552h13m52.067468138s) [seg_pos={0 0}, upload=484167]: Done begin upload segment.
Dec 28 15:42:52.196410110 (552h13m52.067493145s) [noise=true , address=145.79.227.47:4267 , piece_id=2ZQ4TJASHFXYK2XGGKPD67FA5Y7Q5STRKKTKPCEPO6IRFQHYSCBQ, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.196618417 (552h13m52.067701452s) [noise=true , address=77.90.31.11:4554 , piece_id=JTCAZ7VIIIIPRW6VBFCKPBGZWZNL4LUESOY2NEMPPSHWVC7A6ZMQ, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.197061621 (552h13m52.068144666s) [noise=true , address=45.91.116.9:4262 , piece_id=COKORQ2MVWK5Y5HJVBYOGXPYLK5OLKPKVIK46LH2FDLHIUULHX4A, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.197404124 (552h13m52.068487159s) [noise=true , address=45.91.116.13:4155 , piece_id=NTHJNIZVYTRSAUQY2CJLRTQJBQ7XQN7L4G2EOLQKBNG34C2GKYQA, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.197693554 (552h13m52.068776589s) [noise=true , address=145.79.227.15:4874 , piece_id=AP6PWBAKDIXWTFYBLCAONT3SAX7ULXUH2S6A5A4SLSKIBFUQEJRQ, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.198044422 (552h13m52.069127468s) [noise=true , address=145.79.227.41:4087 , piece_id=3T63FN7KCO6HWW3TWANWMWZVCX3R5MPKLBBYWHLRHPIO6X4XHWJA, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.198441750 (552h13m52.069524785s) [noise=true , address=145.79.227.40:4050 , piece_id=FUMVUWO7BOWRJG6Y773SQXI6RBX4TVQY776XPOYCO2MQPQHO3JRA, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.198783940 (552h13m52.069866975s) [noise=true , address=145.79.227.11:4729 , piece_id=CYCWWZINYMH24SLMOCRPRRQCLNMAWUN3KYR4FZ73D4W5OMPA7LHQ, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.200304386 (552h13m52.071387421s) [noise=true , address=145.79.227.12:4765 , piece_id=FP7DDOPFF2WYKPORLDDUDES4UPMOZSUWYLYSJ6BFH663MGJDINIA, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.200354281 (552h13m52.071437317s) [noise=true , address=145.79.227.6:4258 , piece_id=3XQ4C5NJGWMHP7JNRXADWXSYOWWQWBNGVYFWQLTABQ44SGAFNUNA, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.201579464 (552h13m52.072662499s) [noise=true , address=45.91.116.6:4975 , piece_id=PDPMJPLQESWKZSEVSFKELJZ57JWIT3WPB7FVMLWVBAAZHJTTIK4Q, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.202924465 (552h13m52.0740075s) [noise=true , address=77.90.31.13:4627 , piece_id=ISYR5TOM25OKARC2RPH7OSR5IGCJRSHJRLLYYM7C2SE3DRXMPWGQ, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.203999402 (552h13m52.075082437s) [noise=true , address=145.79.227.3:4149 , piece_id=MTDSX6BWEXVIC42OZPN7QCFZAUIHMSVKJCTOZHZPQEXZLWKZAARQ, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.205116268 (552h13m52.076199303s) [noise=true , address=77.90.31.12:4590 , piece_id=ELUEOCZKHHWH2U6YE3L4WGF57GIZ3PIGI46NSY5DWY3XDVDSQYDQ, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.206324519 (552h13m52.077407554s) [noise=true , address=145.79.227.48:4301 , piece_id=4P56GUSVLRESPSO2VHT4YGHPL62JBY4H45LMEQAEQGJ7FCYJFKEQ, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.207960545 (552h13m52.07904358s) [noise=true , address=45.91.116.11:4118 , piece_id=CDCXDDCDEXPDHQHQHSGFWEKAYV7CADTNM7T2MWFLBYRCFVE3OFGQ, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:53.821102547 (552h13m53.692185582s) [upload=484167]: Next returned nil segment
Dec 28 15:42:53.821115652 (552h13m53.692198687s) [upload=484167]: Waiting for error group managing segments...
Dec 28 15:42:54.206082653 (552h13m54.077165699s) [noise=true , address=145.79.227.11:4729 , piece_id=CYCWWZINYMH24SLMOCRPRRQCLNMAWUN3KYR4FZ73D4W5OMPA7LHQ, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:54.209008024 (552h13m54.080091059s) [noise=true , address=145.79.227.3:4149 , piece_id=MTDSX6BWEXVIC42OZPN7QCFZAUIHMSVKJCTOZHZPQEXZLWKZAARQ, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:54.215641706 (552h13m54.086724751s) [noise=true , address=145.79.227.41:4087 , piece_id=3T63FN7KCO6HWW3TWANWMWZVCX3R5MPKLBBYWHLRHPIO6X4XHWJA, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:54.215664429 (552h13m54.086747464s) [noise=true , address=145.79.227.48:4301 , piece_id=4P56GUSVLRESPSO2VHT4YGHPL62JBY4H45LMEQAEQGJ7FCYJFKEQ, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:54.218028180 (552h13m54.089111215s) [noise=true , address=145.79.227.6:4258 , piece_id=3XQ4C5NJGWMHP7JNRXADWXSYOWWQWBNGVYFWQLTABQ44SGAFNUNA, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:54.225929916 (552h13m54.097012951s) [noise=true , address=145.79.227.47:4267 , piece_id=2ZQ4TJASHFXYK2XGGKPD67FA5Y7Q5STRKKTKPCEPO6IRFQHYSCBQ, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:54.226846571 (552h13m54.097929616s) [noise=true , address=145.79.227.15:4874 , piece_id=AP6PWBAKDIXWTFYBLCAONT3SAX7ULXUH2S6A5A4SLSKIBFUQEJRQ, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:54.228507714 (552h13m54.09959076s) [noise=true , address=145.79.227.12:4765 , piece_id=FP7DDOPFF2WYKPORLDDUDES4UPMOZSUWYLYSJ6BFH663MGJDINIA, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:54.470490401 (552h13m54.341573446s) [noise=true , address=45.91.116.6:4975 , piece_id=PDPMJPLQESWKZSEVSFKELJZ57JWIT3WPB7FVMLWVBAAZHJTTIK4Q, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:54.522443228 (552h13m54.393526273s) [noise=true , address=77.90.31.13:4627 , piece_id=ISYR5TOM25OKARC2RPH7OSR5IGCJRSHJRLLYYM7C2SE3DRXMPWGQ, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:54.729445324 (552h13m54.600528359s) [noise=true , address=45.91.116.11:4118 , piece_id=CDCXDDCDEXPDHQHQHSGFWEKAYV7CADTNM7T2MWFLBYRCFVE3OFGQ, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:55.259436045 (552h13m55.13051908s) [noise=true , address=45.91.116.9:4262 , piece_id=COKORQ2MVWK5Y5HJVBYOGXPYLK5OLKPKVIK46LH2FDLHIUULHX4A, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:55.535357185 (552h13m55.40644022s) [noise=true , address=77.90.31.11:4554 , piece_id=JTCAZ7VIIIIPRW6VBFCKPBGZWZNL4LUESOY2NEMPPSHWVC7A6ZMQ, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:55.535380219 (552h13m55.406463274s) [seg_pos={0 0}, upload=484167]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:42:55.535624404 (552h13m55.406707459s) [noise=true , address=45.91.116.13:4155 , piece_id=NTHJNIZVYTRSAUQY2CJLRTQJBQ7XQN7L4G2EOLQKBNG34C2GKYQA, seg_pos={0 0}, upload=484167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 28 15:42:55.535634243 (552h13m55.406717298s) [noise=true , address=45.91.116.13:4155 , piece_id=NTHJNIZVYTRSAUQY2CJLRTQJBQ7XQN7L4G2EOLQKBNG34C2GKYQA, seg_pos={0 0}, upload=484167]: Long tail cancellation due to optimal threshold
Dec 28 15:42:55.535647808 (552h13m55.406730853s) [noise=true , address=77.90.31.12:4590 , piece_id=ELUEOCZKHHWH2U6YE3L4WGF57GIZ3PIGI46NSY5DWY3XDVDSQYDQ, seg_pos={0 0}, upload=484167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 28 15:42:55.535657025 (552h13m55.40674007s) [noise=true , address=77.90.31.12:4590 , piece_id=ELUEOCZKHHWH2U6YE3L4WGF57GIZ3PIGI46NSY5DWY3XDVDSQYDQ, seg_pos={0 0}, upload=484167]: Long tail cancellation due to optimal threshold
Dec 28 15:42:55.535895239 (552h13m55.406978274s) [noise=true , address=145.79.227.40:4050 , piece_id=FUMVUWO7BOWRJG6Y773SQXI6RBX4TVQY776XPOYCO2MQPQHO3JRA, seg_pos={0 0}, upload=484167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF
Dec 28 15:42:55.535904407 (552h13m55.406987442s) [noise=true , address=145.79.227.40:4050 , piece_id=FUMVUWO7BOWRJG6Y773SQXI6RBX4TVQY776XPOYCO2MQPQHO3JRA, seg_pos={0 0}, upload=484167]: Long tail cancellation due to optimal threshold
Dec 28 15:42:55.535927881 (552h13m55.407010916s) [seg_pos={0 0}, upload=484167]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:42:55.536025407 (552h13m55.407108452s) [upload=484167]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:42:55.536032120 (552h13m55.407115165s) [upload=484167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:42:55.831843836 (552h13m55.702926871s) [upload=484167]: Done uploading segments...
Dec 28 15:42:55.831861769 (552h13m55.702944814s) [upload=484167]: Upload finished. err: <nil>
Duration: 6.241430928s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..XX...XX] duration:
5.001243405s trace:
https://jaeger.dev.storj.io/trace/5429a4e7f4ecc79
FAIL (2025-12-28 15:43:46 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00126678s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
12.800303332s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 15:43:46 UTC)
orbiter warning: context deadline exceeded
Duration: 12.800330773s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................X................................X.........] duration:
14.876444573s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:43:38 UTC)
orbiter warning: context deadline exceeded
Duration: 14.876472085s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [.....X.X.X.....X......XX.X..XXX.X.XXX.XX.XX...X.X........X.X] duration:
30.003954994s trace:
https://jaeger.dev.storj.io/trace/3862ec468111a894
FAIL (2025-12-28 15:43:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: PDPMJPLQESWKZSEVSFKELJZ57JWIT3WPB7FVMLWVBAAZHJTTIK4Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: COKORQ2MVWK5Y5HJVBYOGXPYLK5OLKPKVIK46LH2FDLHIUULHX4A) context deadline exceeded
======LOGS======
Dec 28 15:42:51.947870419 (552h13m51.818953454s) [upload=484167]: Starting upload
Dec 28 15:42:51.947931585 (552h13m51.81901464s) [upload=484167]: Done starting upload
Dec 28 15:42:51.948034371 (552h13m51.819117416s) [upload=484167]: Uploading segments...
Dec 28 15:42:51.948157325 (552h13m51.81924036s) [upload=484167]: Got next segment. Inline: false
Dec 28 15:42:51.948214774 (552h13m51.81929781s) [upload=484167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:42:51.948221377 (552h13m51.819304412s) [upload=484167]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:42:52.196339967 (552h13m52.067423002s) [seg_pos={0 0}, upload=484167]: Begin upload segment...
Dec 28 15:42:52.196385092 (552h13m52.067468138s) [seg_pos={0 0}, upload=484167]: Done begin upload segment.
Dec 28 15:42:52.196410110 (552h13m52.067493145s) [noise=true , address=145.79.227.47:4267 , piece_id=2ZQ4TJASHFXYK2XGGKPD67FA5Y7Q5STRKKTKPCEPO6IRFQHYSCBQ, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.196618417 (552h13m52.067701452s) [noise=true , address=77.90.31.11:4554 , piece_id=JTCAZ7VIIIIPRW6VBFCKPBGZWZNL4LUESOY2NEMPPSHWVC7A6ZMQ, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.197061621 (552h13m52.068144666s) [noise=true , address=45.91.116.9:4262 , piece_id=COKORQ2MVWK5Y5HJVBYOGXPYLK5OLKPKVIK46LH2FDLHIUULHX4A, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.197404124 (552h13m52.068487159s) [noise=true , address=45.91.116.13:4155 , piece_id=NTHJNIZVYTRSAUQY2CJLRTQJBQ7XQN7L4G2EOLQKBNG34C2GKYQA, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.197693554 (552h13m52.068776589s) [noise=true , address=145.79.227.15:4874 , piece_id=AP6PWBAKDIXWTFYBLCAONT3SAX7ULXUH2S6A5A4SLSKIBFUQEJRQ, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.198044422 (552h13m52.069127468s) [noise=true , address=145.79.227.41:4087 , piece_id=3T63FN7KCO6HWW3TWANWMWZVCX3R5MPKLBBYWHLRHPIO6X4XHWJA, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.198441750 (552h13m52.069524785s) [noise=true , address=145.79.227.40:4050 , piece_id=FUMVUWO7BOWRJG6Y773SQXI6RBX4TVQY776XPOYCO2MQPQHO3JRA, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.198783940 (552h13m52.069866975s) [noise=true , address=145.79.227.11:4729 , piece_id=CYCWWZINYMH24SLMOCRPRRQCLNMAWUN3KYR4FZ73D4W5OMPA7LHQ, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.200304386 (552h13m52.071387421s) [noise=true , address=145.79.227.12:4765 , piece_id=FP7DDOPFF2WYKPORLDDUDES4UPMOZSUWYLYSJ6BFH663MGJDINIA, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.200354281 (552h13m52.071437317s) [noise=true , address=145.79.227.6:4258 , piece_id=3XQ4C5NJGWMHP7JNRXADWXSYOWWQWBNGVYFWQLTABQ44SGAFNUNA, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.201579464 (552h13m52.072662499s) [noise=true , address=45.91.116.6:4975 , piece_id=PDPMJPLQESWKZSEVSFKELJZ57JWIT3WPB7FVMLWVBAAZHJTTIK4Q, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.202924465 (552h13m52.0740075s) [noise=true , address=77.90.31.13:4627 , piece_id=ISYR5TOM25OKARC2RPH7OSR5IGCJRSHJRLLYYM7C2SE3DRXMPWGQ, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.203999402 (552h13m52.075082437s) [noise=true , address=145.79.227.3:4149 , piece_id=MTDSX6BWEXVIC42OZPN7QCFZAUIHMSVKJCTOZHZPQEXZLWKZAARQ, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.205116268 (552h13m52.076199303s) [noise=true , address=77.90.31.12:4590 , piece_id=ELUEOCZKHHWH2U6YE3L4WGF57GIZ3PIGI46NSY5DWY3XDVDSQYDQ, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.206324519 (552h13m52.077407554s) [noise=true , address=145.79.227.48:4301 , piece_id=4P56GUSVLRESPSO2VHT4YGHPL62JBY4H45LMEQAEQGJ7FCYJFKEQ, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:52.207960545 (552h13m52.07904358s) [noise=true , address=45.91.116.11:4118 , piece_id=CDCXDDCDEXPDHQHQHSGFWEKAYV7CADTNM7T2MWFLBYRCFVE3OFGQ, seg_pos={0 0}, upload=484167]: Uploading piece...
Dec 28 15:42:53.821102547 (552h13m53.692185582s) [upload=484167]: Next returned nil segment
Dec 28 15:42:53.821115652 (552h13m53.692198687s) [upload=484167]: Waiting for error group managing segments...
Dec 28 15:42:54.206082653 (552h13m54.077165699s) [noise=true , address=145.79.227.11:4729 , piece_id=CYCWWZINYMH24SLMOCRPRRQCLNMAWUN3KYR4FZ73D4W5OMPA7LHQ, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:54.209008024 (552h13m54.080091059s) [noise=true , address=145.79.227.3:4149 , piece_id=MTDSX6BWEXVIC42OZPN7QCFZAUIHMSVKJCTOZHZPQEXZLWKZAARQ, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:54.215641706 (552h13m54.086724751s) [noise=true , address=145.79.227.41:4087 , piece_id=3T63FN7KCO6HWW3TWANWMWZVCX3R5MPKLBBYWHLRHPIO6X4XHWJA, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:54.215664429 (552h13m54.086747464s) [noise=true , address=145.79.227.48:4301 , piece_id=4P56GUSVLRESPSO2VHT4YGHPL62JBY4H45LMEQAEQGJ7FCYJFKEQ, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:54.218028180 (552h13m54.089111215s) [noise=true , address=145.79.227.6:4258 , piece_id=3XQ4C5NJGWMHP7JNRXADWXSYOWWQWBNGVYFWQLTABQ44SGAFNUNA, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:54.225929916 (552h13m54.097012951s) [noise=true , address=145.79.227.47:4267 , piece_id=2ZQ4TJASHFXYK2XGGKPD67FA5Y7Q5STRKKTKPCEPO6IRFQHYSCBQ, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:54.226846571 (552h13m54.097929616s) [noise=true , address=145.79.227.15:4874 , piece_id=AP6PWBAKDIXWTFYBLCAONT3SAX7ULXUH2S6A5A4SLSKIBFUQEJRQ, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:54.228507714 (552h13m54.09959076s) [noise=true , address=145.79.227.12:4765 , piece_id=FP7DDOPFF2WYKPORLDDUDES4UPMOZSUWYLYSJ6BFH663MGJDINIA, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:54.470490401 (552h13m54.341573446s) [noise=true , address=45.91.116.6:4975 , piece_id=PDPMJPLQESWKZSEVSFKELJZ57JWIT3WPB7FVMLWVBAAZHJTTIK4Q, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:54.522443228 (552h13m54.393526273s) [noise=true , address=77.90.31.13:4627 , piece_id=ISYR5TOM25OKARC2RPH7OSR5IGCJRSHJRLLYYM7C2SE3DRXMPWGQ, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:54.729445324 (552h13m54.600528359s) [noise=true , address=45.91.116.11:4118 , piece_id=CDCXDDCDEXPDHQHQHSGFWEKAYV7CADTNM7T2MWFLBYRCFVE3OFGQ, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:55.259436045 (552h13m55.13051908s) [noise=true , address=45.91.116.9:4262 , piece_id=COKORQ2MVWK5Y5HJVBYOGXPYLK5OLKPKVIK46LH2FDLHIUULHX4A, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:55.535357185 (552h13m55.40644022s) [noise=true , address=77.90.31.11:4554 , piece_id=JTCAZ7VIIIIPRW6VBFCKPBGZWZNL4LUESOY2NEMPPSHWVC7A6ZMQ, seg_pos={0 0}, upload=484167]: Done uploading piece. err: <nil>
Dec 28 15:42:55.535380219 (552h13m55.406463274s) [seg_pos={0 0}, upload=484167]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:42:55.535624404 (552h13m55.406707459s) [noise=true , address=45.91.116.13:4155 , piece_id=NTHJNIZVYTRSAUQY2CJLRTQJBQ7XQN7L4G2EOLQKBNG34C2GKYQA, seg_pos={0 0}, upload=484167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 28 15:42:55.535634243 (552h13m55.406717298s) [noise=true , address=45.91.116.13:4155 , piece_id=NTHJNIZVYTRSAUQY2CJLRTQJBQ7XQN7L4G2EOLQKBNG34C2GKYQA, seg_pos={0 0}, upload=484167]: Long tail cancellation due to optimal threshold
Dec 28 15:42:55.535647808 (552h13m55.406730853s) [noise=true , address=77.90.31.12:4590 , piece_id=ELUEOCZKHHWH2U6YE3L4WGF57GIZ3PIGI46NSY5DWY3XDVDSQYDQ, seg_pos={0 0}, upload=484167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 28 15:42:55.535657025 (552h13m55.40674007s) [noise=true , address=77.90.31.12:4590 , piece_id=ELUEOCZKHHWH2U6YE3L4WGF57GIZ3PIGI46NSY5DWY3XDVDSQYDQ, seg_pos={0 0}, upload=484167]: Long tail cancellation due to optimal threshold
Dec 28 15:42:55.535895239 (552h13m55.406978274s) [noise=true , address=145.79.227.40:4050 , piece_id=FUMVUWO7BOWRJG6Y773SQXI6RBX4TVQY776XPOYCO2MQPQHO3JRA, seg_pos={0 0}, upload=484167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF
Dec 28 15:42:55.535904407 (552h13m55.406987442s) [noise=true , address=145.79.227.40:4050 , piece_id=FUMVUWO7BOWRJG6Y773SQXI6RBX4TVQY776XPOYCO2MQPQHO3JRA, seg_pos={0 0}, upload=484167]: Long tail cancellation due to optimal threshold
Dec 28 15:42:55.535927881 (552h13m55.407010916s) [seg_pos={0 0}, upload=484167]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:42:55.536025407 (552h13m55.407108452s) [upload=484167]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:42:55.536032120 (552h13m55.407115165s) [upload=484167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:42:55.831843836 (552h13m55.702926871s) [upload=484167]: Done uploading segments...
Dec 28 15:42:55.831861769 (552h13m55.702944814s) [upload=484167]: Upload finished. err: <nil>
Duration: 30.003976315s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......................X....X...X........................X..] duration:
45.895755244s trace:
https://jaeger.dev.storj.io/trace/3cc795a000779b86
FAIL (2025-12-28 15:43:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ) context deadline exceeded; piecestore: (Node ID: 128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, Piece ID: GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q) context deadline exceeded; piecestore: (Node ID: 12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, Piece ID: TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A) context deadline exceeded
======LOGS======
48.156469147 (552h11m48.027552193s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157156416 (552h11m48.028239461s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157556748 (552h11m48.028639793s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157945468 (552h11m48.029028503s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158645151 (552h11m48.029728196s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158842136 (552h11m48.029925171s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158983275 (552h11m48.03006631s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159344152 (552h11m48.030427187s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159350725 (552h11m48.03043376s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159710760 (552h11m48.030793795s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159732391 (552h11m48.030815437s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160068310 (552h11m48.031151346s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160439438 (552h11m48.031522473s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160813179 (552h11m48.031896225s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160839249 (552h11m48.031922284s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161271813 (552h11m48.032354848s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161765813 (552h11m48.032848848s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:49.456221594 (552h11m49.327304629s) [upload=484135]: Next returned nil segment
Dec 28 15:40:49.456235460 (552h11m49.327318495s) [upload=484135]: Waiting for error group managing segments...
Dec 28 15:40:49.789381987 (552h11m49.660465032s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.818159755 (552h11m49.6892428s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.901730607 (552h11m49.772813642s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.297735267 (552h11m50.168818312s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.866598958 (552h11m50.737682003s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.955467169 (552h11m50.826550214s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.029562809 (552h11m50.900645854s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.033202229 (552h11m50.904285264s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.119305864 (552h11m50.990388909s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.271054029 (552h11m51.142137074s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.374827580 (552h11m51.245910626s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.380607978 (552h11m51.251691013s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.500440018 (552h11m51.371523063s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688250240 (552h11m51.559333285s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688275438 (552h11m51.559358473s) [seg_pos={0 0}, upload=484135]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:40:51.688501768 (552h11m51.559584804s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688512109 (552h11m51.559595144s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688551694 (552h11m51.559634729s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688560741 (552h11m51.559643786s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688768767 (552h11m51.559851802s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWqAGD7Vrj2yzEVyNbfPp278Gga42xPLyAcdzGvUSuz3EgJYh): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688776031 (552h11m51.559859066s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688811729 (552h11m51.559894784s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688820707 (552h11m51.559903752s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688840654 (552h11m51.559923689s) [seg_pos={0 0}, upload=484135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:40:51.688958800 (552h11m51.560041845s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:40:51.688965582 (552h11m51.560048617s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:40:51.990040599 (552h11m51.861123664s) [upload=484135]: Done uploading segments...
Dec 28 15:40:51.990091757 (552h11m51.861174802s) [upload=484135]: Upload finished. err: <nil>
Duration: 45.895772567s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................X................................X........] duration:
18.132339511s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:43:15 UTC)
orbiter warning: context deadline exceeded
Duration: 18.132362364s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..XX...X] duration:
5.001292268s trace:
https://jaeger.dev.storj.io/trace/159a5039059cab7e
FAIL (2025-12-28 15:42:51 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001310251s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [......X.X.X.....X......XX.X..XXX.X.XXX.XX.XX...X.X........X.] duration:
5.838448544s trace:
https://jaeger.dev.storj.io/trace/42a48e5bbb73007f
FAIL (2025-12-28 15:42:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 15:42:04.670653653 (552h13m4.541736698s) [upload=484155]: Starting upload
Dec 28 15:42:04.670723146 (552h13m4.541806181s) [upload=484155]: Done starting upload
Dec 28 15:42:04.670771148 (552h13m4.541854203s) [upload=484155]: Uploading segments...
Dec 28 15:42:04.670969064 (552h13m4.542052099s) [upload=484155]: Got next segment. Inline: false
Dec 28 15:42:04.671009320 (552h13m4.542092355s) [upload=484155]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:42:04.671014701 (552h13m4.542097737s) [upload=484155]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:42:04.916063766 (552h13m4.787146801s) [seg_pos={0 0}, upload=484155]: Begin upload segment...
Dec 28 15:42:04.916114523 (552h13m4.787197558s) [seg_pos={0 0}, upload=484155]: Done begin upload segment.
Dec 28 15:42:04.916147416 (552h13m4.787230451s) [noise=true , address=145.79.227.15:4874 , piece_id=5ZSHIRBICZDITRYHC2WRSKI2D3N3FAGLRMZTYQM3Z5HVF7NYYVUQ, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.916160941 (552h13m4.787243986s) [noise=true , address=145.79.227.47:4266 , piece_id=UJEKJM56JUKLPYL7XCRVN5IRMM3I2WN552D2DSIROLIVTGBXIBWA, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.916195497 (552h13m4.787278532s) [noise=true , address=145.79.227.8:4330 , piece_id=ZNL57QAFQA6YFCQEUPMIRQQ7IPGS6IDIZYGGYVJCR6C7O6VDFS3Q, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.916537418 (552h13m4.787620463s) [noise=true , address=77.90.31.17:4770 , piece_id=LONUP6HABDAOSFH5XCWLAPEVQRP7GW6RCV6AHJPQAWNV3CFZ3SWQ, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.917874203 (552h13m4.788957248s) [noise=true , address=77.90.31.10:4518 , piece_id=BO2SUOP6PQ2VVVYLRP6C3JMIRZQUNTDDFAL5A772J2DMIBKKKIRQ, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.918256621 (552h13m4.789339666s) [noise=true , address=77.90.31.14:4663 , piece_id=7CSTTEXLKULZLD5EYGK2IUZUFK4XDKRK3DARKEK3APAAO6GSHWBA, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.918749700 (552h13m4.789832735s) [noise=true , address=145.79.227.48:4303 , piece_id=NW3CYEMDMGSVO56LD4BYXMVHGX6D7KD4XLF54KOE7WJHLU45XZHQ, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.919697044 (552h13m4.790780079s) [noise=true , address=145.79.227.17:4971 , piece_id=AHOA3PTFYSFFZ5JWYEUGDRFJRSHQTSLHZVLB7KDPTEWQLZ6V6M7Q, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.919886926 (552h13m4.790969971s) [noise=true , address=77.90.31.16:4736 , piece_id=7GCHSVXYBMXUORZO24QRFQWTYMLHIJX74CADH3WCRJASYCCVCTVQ, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.920804663 (552h13m4.791887698s) [noise=true , address=145.79.227.19:4079 , piece_id=PSEWYVNEJPO4TQ64ALTGY6TELXYW644GC3LFKNAZZIYCBVZV2X7A, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.921042827 (552h13m4.792125862s) [noise=true , address=145.79.227.13:4802 , piece_id=LRDBURULOYHYN7DHXCOMNK3CPJ7FL24T6JFYBR6HX3WQSTGXBUNQ, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.922273530 (552h13m4.793356565s) [noise=true , address=45.91.116.10:4084 , piece_id=UG4NODPA2OLOFXIR3NJWEJTHKVRQ3IUEWKI7GTWIVY3NJHV2KGQQ, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.923799225 (552h13m4.79488227s) [noise=true , address=145.79.227.50:4375 , piece_id=3LEIEK6HUT7LMI3KX4GSRXQNVERYO34BPG7CPHVVF535EKOTRDLA, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.925162190 (552h13m4.796245225s) [noise=true , address=45.91.116.9:4262 , piece_id=HY3PMZELGWUVDXIBWGT2HMWXPO2YFWREKRYCLD4ZTXHOIDH3RU7Q, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.926518042 (552h13m4.797601077s) [noise=true , address=145.79.227.51:4409 , piece_id=4SV32A5QPQJ23LIBIEZJMQICP6EXHFYH4NH6ZZ355K6C25CW3PPA, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.926660223 (552h13m4.797743258s) [noise=true , address=45.91.116.6:4975 , piece_id=QPN66274VV5WREFJS45QFJ6LCSBSBBEVJRMDUEH6ZZMWYDFKXT5A, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:06.534590149 (552h13m6.405673184s) [upload=484155]: Next returned nil segment
Dec 28 15:42:06.534607001 (552h13m6.405690046s) [upload=484155]: Waiting for error group managing segments...
Dec 28 15:42:06.916569781 (552h13m6.787652827s) [noise=true , address=145.79.227.19:4079 , piece_id=PSEWYVNEJPO4TQ64ALTGY6TELXYW644GC3LFKNAZZIYCBVZV2X7A, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:06.920941256 (552h13m6.792024291s) [noise=true , address=145.79.227.15:4874 , piece_id=5ZSHIRBICZDITRYHC2WRSKI2D3N3FAGLRMZTYQM3Z5HVF7NYYVUQ, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:06.923538572 (552h13m6.794621617s) [noise=true , address=145.79.227.8:4330 , piece_id=ZNL57QAFQA6YFCQEUPMIRQQ7IPGS6IDIZYGGYVJCR6C7O6VDFS3Q, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:06.929891960 (552h13m6.800974995s) [noise=true , address=145.79.227.13:4802 , piece_id=LRDBURULOYHYN7DHXCOMNK3CPJ7FL24T6JFYBR6HX3WQSTGXBUNQ, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:06.932495537 (552h13m6.803578582s) [noise=true , address=145.79.227.48:4303 , piece_id=NW3CYEMDMGSVO56LD4BYXMVHGX6D7KD4XLF54KOE7WJHLU45XZHQ, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:06.934270778 (552h13m6.805353813s) [noise=true , address=145.79.227.47:4266 , piece_id=UJEKJM56JUKLPYL7XCRVN5IRMM3I2WN552D2DSIROLIVTGBXIBWA, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:06.936146439 (552h13m6.807229485s) [noise=true , address=145.79.227.50:4375 , piece_id=3LEIEK6HUT7LMI3KX4GSRXQNVERYO34BPG7CPHVVF535EKOTRDLA, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:06.940747430 (552h13m6.811830465s) [noise=true , address=145.79.227.17:4971 , piece_id=AHOA3PTFYSFFZ5JWYEUGDRFJRSHQTSLHZVLB7KDPTEWQLZ6V6M7Q, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:06.946135420 (552h13m6.817218465s) [noise=true , address=145.79.227.51:4409 , piece_id=4SV32A5QPQJ23LIBIEZJMQICP6EXHFYH4NH6ZZ355K6C25CW3PPA, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:07.092411053 (552h13m6.963494098s) [noise=true , address=77.90.31.14:4663 , piece_id=7CSTTEXLKULZLD5EYGK2IUZUFK4XDKRK3DARKEK3APAAO6GSHWBA, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:07.143026113 (552h13m7.014109158s) [noise=true , address=45.91.116.6:4975 , piece_id=QPN66274VV5WREFJS45QFJ6LCSBSBBEVJRMDUEH6ZZMWYDFKXT5A, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:07.174202007 (552h13m7.045285042s) [noise=true , address=45.91.116.10:4084 , piece_id=UG4NODPA2OLOFXIR3NJWEJTHKVRQ3IUEWKI7GTWIVY3NJHV2KGQQ, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:07.202427804 (552h13m7.073510869s) [noise=true , address=77.90.31.10:4518 , piece_id=BO2SUOP6PQ2VVVYLRP6C3JMIRZQUNTDDFAL5A772J2DMIBKKKIRQ, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:07.202454775 (552h13m7.07353782s) [seg_pos={0 0}, upload=484155]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:42:07.202661067 (552h13m7.073744102s) [noise=true , address=45.91.116.9:4262 , piece_id=HY3PMZELGWUVDXIBWGT2HMWXPO2YFWREKRYCLD4ZTXHOIDH3RU7Q, seg_pos={0 0}, upload=484155]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF
Dec 28 15:42:07.202668172 (552h13m7.073751207s) [noise=true , address=45.91.116.9:4262 , piece_id=HY3PMZELGWUVDXIBWGT2HMWXPO2YFWREKRYCLD4ZTXHOIDH3RU7Q, seg_pos={0 0}, upload=484155]: Long tail cancellation due to optimal threshold
Dec 28 15:42:07.202708368 (552h13m7.073791413s) [noise=true , address=77.90.31.16:4736 , piece_id=7GCHSVXYBMXUORZO24QRFQWTYMLHIJX74CADH3WCRJASYCCVCTVQ, seg_pos={0 0}, upload=484155]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 28 15:42:07.202800272 (552h13m7.073883317s) [noise=true , address=77.90.31.16:4736 , piece_id=7GCHSVXYBMXUORZO24QRFQWTYMLHIJX74CADH3WCRJASYCCVCTVQ, seg_pos={0 0}, upload=484155]: Long tail cancellation due to optimal threshold
Dec 28 15:42:07.202896476 (552h13m7.073979511s) [noise=true , address=77.90.31.17:4770 , piece_id=LONUP6HABDAOSFH5XCWLAPEVQRP7GW6RCV6AHJPQAWNV3CFZ3SWQ, seg_pos={0 0}, upload=484155]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:40228->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:40228->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40228->77.90.31.17:4770: use of closed network connection
Dec 28 15:42:07.202903349 (552h13m7.073986384s) [noise=true , address=77.90.31.17:4770 , piece_id=LONUP6HABDAOSFH5XCWLAPEVQRP7GW6RCV6AHJPQAWNV3CFZ3SWQ, seg_pos={0 0}, upload=484155]: Long tail cancellation due to optimal threshold
Dec 28 15:42:07.202918859 (552h13m7.074001894s) [seg_pos={0 0}, upload=484155]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:42:07.202989804 (552h13m7.074072849s) [upload=484155]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:42:07.202995174 (552h13m7.074078209s) [upload=484155]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:42:07.496201678 (552h13m7.367284713s) [upload=484155]: Done uploading segments...
Dec 28 15:42:07.496221165 (552h13m7.36730421s) [upload=484155]: Upload finished. err: <nil>
Duration: 5.838466577s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
5.895859455s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 15:42:36 UTC)
orbiter warning: context deadline exceeded
Duration: 5.895886997s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [X......X.X.X.....X......XX.X..XXX.X.XXX.XX.XX...X.X........X] duration:
30.003172614s trace:
https://jaeger.dev.storj.io/trace/42a48e5bbb73007f
FAIL (2025-12-28 15:42:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 15:42:04.670653653 (552h13m4.541736698s) [upload=484155]: Starting upload
Dec 28 15:42:04.670723146 (552h13m4.541806181s) [upload=484155]: Done starting upload
Dec 28 15:42:04.670771148 (552h13m4.541854203s) [upload=484155]: Uploading segments...
Dec 28 15:42:04.670969064 (552h13m4.542052099s) [upload=484155]: Got next segment. Inline: false
Dec 28 15:42:04.671009320 (552h13m4.542092355s) [upload=484155]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:42:04.671014701 (552h13m4.542097737s) [upload=484155]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:42:04.916063766 (552h13m4.787146801s) [seg_pos={0 0}, upload=484155]: Begin upload segment...
Dec 28 15:42:04.916114523 (552h13m4.787197558s) [seg_pos={0 0}, upload=484155]: Done begin upload segment.
Dec 28 15:42:04.916147416 (552h13m4.787230451s) [noise=true , address=145.79.227.15:4874 , piece_id=5ZSHIRBICZDITRYHC2WRSKI2D3N3FAGLRMZTYQM3Z5HVF7NYYVUQ, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.916160941 (552h13m4.787243986s) [noise=true , address=145.79.227.47:4266 , piece_id=UJEKJM56JUKLPYL7XCRVN5IRMM3I2WN552D2DSIROLIVTGBXIBWA, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.916195497 (552h13m4.787278532s) [noise=true , address=145.79.227.8:4330 , piece_id=ZNL57QAFQA6YFCQEUPMIRQQ7IPGS6IDIZYGGYVJCR6C7O6VDFS3Q, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.916537418 (552h13m4.787620463s) [noise=true , address=77.90.31.17:4770 , piece_id=LONUP6HABDAOSFH5XCWLAPEVQRP7GW6RCV6AHJPQAWNV3CFZ3SWQ, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.917874203 (552h13m4.788957248s) [noise=true , address=77.90.31.10:4518 , piece_id=BO2SUOP6PQ2VVVYLRP6C3JMIRZQUNTDDFAL5A772J2DMIBKKKIRQ, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.918256621 (552h13m4.789339666s) [noise=true , address=77.90.31.14:4663 , piece_id=7CSTTEXLKULZLD5EYGK2IUZUFK4XDKRK3DARKEK3APAAO6GSHWBA, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.918749700 (552h13m4.789832735s) [noise=true , address=145.79.227.48:4303 , piece_id=NW3CYEMDMGSVO56LD4BYXMVHGX6D7KD4XLF54KOE7WJHLU45XZHQ, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.919697044 (552h13m4.790780079s) [noise=true , address=145.79.227.17:4971 , piece_id=AHOA3PTFYSFFZ5JWYEUGDRFJRSHQTSLHZVLB7KDPTEWQLZ6V6M7Q, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.919886926 (552h13m4.790969971s) [noise=true , address=77.90.31.16:4736 , piece_id=7GCHSVXYBMXUORZO24QRFQWTYMLHIJX74CADH3WCRJASYCCVCTVQ, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.920804663 (552h13m4.791887698s) [noise=true , address=145.79.227.19:4079 , piece_id=PSEWYVNEJPO4TQ64ALTGY6TELXYW644GC3LFKNAZZIYCBVZV2X7A, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.921042827 (552h13m4.792125862s) [noise=true , address=145.79.227.13:4802 , piece_id=LRDBURULOYHYN7DHXCOMNK3CPJ7FL24T6JFYBR6HX3WQSTGXBUNQ, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.922273530 (552h13m4.793356565s) [noise=true , address=45.91.116.10:4084 , piece_id=UG4NODPA2OLOFXIR3NJWEJTHKVRQ3IUEWKI7GTWIVY3NJHV2KGQQ, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.923799225 (552h13m4.79488227s) [noise=true , address=145.79.227.50:4375 , piece_id=3LEIEK6HUT7LMI3KX4GSRXQNVERYO34BPG7CPHVVF535EKOTRDLA, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.925162190 (552h13m4.796245225s) [noise=true , address=45.91.116.9:4262 , piece_id=HY3PMZELGWUVDXIBWGT2HMWXPO2YFWREKRYCLD4ZTXHOIDH3RU7Q, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.926518042 (552h13m4.797601077s) [noise=true , address=145.79.227.51:4409 , piece_id=4SV32A5QPQJ23LIBIEZJMQICP6EXHFYH4NH6ZZ355K6C25CW3PPA, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:04.926660223 (552h13m4.797743258s) [noise=true , address=45.91.116.6:4975 , piece_id=QPN66274VV5WREFJS45QFJ6LCSBSBBEVJRMDUEH6ZZMWYDFKXT5A, seg_pos={0 0}, upload=484155]: Uploading piece...
Dec 28 15:42:06.534590149 (552h13m6.405673184s) [upload=484155]: Next returned nil segment
Dec 28 15:42:06.534607001 (552h13m6.405690046s) [upload=484155]: Waiting for error group managing segments...
Dec 28 15:42:06.916569781 (552h13m6.787652827s) [noise=true , address=145.79.227.19:4079 , piece_id=PSEWYVNEJPO4TQ64ALTGY6TELXYW644GC3LFKNAZZIYCBVZV2X7A, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:06.920941256 (552h13m6.792024291s) [noise=true , address=145.79.227.15:4874 , piece_id=5ZSHIRBICZDITRYHC2WRSKI2D3N3FAGLRMZTYQM3Z5HVF7NYYVUQ, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:06.923538572 (552h13m6.794621617s) [noise=true , address=145.79.227.8:4330 , piece_id=ZNL57QAFQA6YFCQEUPMIRQQ7IPGS6IDIZYGGYVJCR6C7O6VDFS3Q, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:06.929891960 (552h13m6.800974995s) [noise=true , address=145.79.227.13:4802 , piece_id=LRDBURULOYHYN7DHXCOMNK3CPJ7FL24T6JFYBR6HX3WQSTGXBUNQ, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:06.932495537 (552h13m6.803578582s) [noise=true , address=145.79.227.48:4303 , piece_id=NW3CYEMDMGSVO56LD4BYXMVHGX6D7KD4XLF54KOE7WJHLU45XZHQ, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:06.934270778 (552h13m6.805353813s) [noise=true , address=145.79.227.47:4266 , piece_id=UJEKJM56JUKLPYL7XCRVN5IRMM3I2WN552D2DSIROLIVTGBXIBWA, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:06.936146439 (552h13m6.807229485s) [noise=true , address=145.79.227.50:4375 , piece_id=3LEIEK6HUT7LMI3KX4GSRXQNVERYO34BPG7CPHVVF535EKOTRDLA, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:06.940747430 (552h13m6.811830465s) [noise=true , address=145.79.227.17:4971 , piece_id=AHOA3PTFYSFFZ5JWYEUGDRFJRSHQTSLHZVLB7KDPTEWQLZ6V6M7Q, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:06.946135420 (552h13m6.817218465s) [noise=true , address=145.79.227.51:4409 , piece_id=4SV32A5QPQJ23LIBIEZJMQICP6EXHFYH4NH6ZZ355K6C25CW3PPA, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:07.092411053 (552h13m6.963494098s) [noise=true , address=77.90.31.14:4663 , piece_id=7CSTTEXLKULZLD5EYGK2IUZUFK4XDKRK3DARKEK3APAAO6GSHWBA, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:07.143026113 (552h13m7.014109158s) [noise=true , address=45.91.116.6:4975 , piece_id=QPN66274VV5WREFJS45QFJ6LCSBSBBEVJRMDUEH6ZZMWYDFKXT5A, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:07.174202007 (552h13m7.045285042s) [noise=true , address=45.91.116.10:4084 , piece_id=UG4NODPA2OLOFXIR3NJWEJTHKVRQ3IUEWKI7GTWIVY3NJHV2KGQQ, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:07.202427804 (552h13m7.073510869s) [noise=true , address=77.90.31.10:4518 , piece_id=BO2SUOP6PQ2VVVYLRP6C3JMIRZQUNTDDFAL5A772J2DMIBKKKIRQ, seg_pos={0 0}, upload=484155]: Done uploading piece. err: <nil>
Dec 28 15:42:07.202454775 (552h13m7.07353782s) [seg_pos={0 0}, upload=484155]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:42:07.202661067 (552h13m7.073744102s) [noise=true , address=45.91.116.9:4262 , piece_id=HY3PMZELGWUVDXIBWGT2HMWXPO2YFWREKRYCLD4ZTXHOIDH3RU7Q, seg_pos={0 0}, upload=484155]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF
Dec 28 15:42:07.202668172 (552h13m7.073751207s) [noise=true , address=45.91.116.9:4262 , piece_id=HY3PMZELGWUVDXIBWGT2HMWXPO2YFWREKRYCLD4ZTXHOIDH3RU7Q, seg_pos={0 0}, upload=484155]: Long tail cancellation due to optimal threshold
Dec 28 15:42:07.202708368 (552h13m7.073791413s) [noise=true , address=77.90.31.16:4736 , piece_id=7GCHSVXYBMXUORZO24QRFQWTYMLHIJX74CADH3WCRJASYCCVCTVQ, seg_pos={0 0}, upload=484155]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 28 15:42:07.202800272 (552h13m7.073883317s) [noise=true , address=77.90.31.16:4736 , piece_id=7GCHSVXYBMXUORZO24QRFQWTYMLHIJX74CADH3WCRJASYCCVCTVQ, seg_pos={0 0}, upload=484155]: Long tail cancellation due to optimal threshold
Dec 28 15:42:07.202896476 (552h13m7.073979511s) [noise=true , address=77.90.31.17:4770 , piece_id=LONUP6HABDAOSFH5XCWLAPEVQRP7GW6RCV6AHJPQAWNV3CFZ3SWQ, seg_pos={0 0}, upload=484155]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:40228->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:40228->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40228->77.90.31.17:4770: use of closed network connection
Dec 28 15:42:07.202903349 (552h13m7.073986384s) [noise=true , address=77.90.31.17:4770 , piece_id=LONUP6HABDAOSFH5XCWLAPEVQRP7GW6RCV6AHJPQAWNV3CFZ3SWQ, seg_pos={0 0}, upload=484155]: Long tail cancellation due to optimal threshold
Dec 28 15:42:07.202918859 (552h13m7.074001894s) [seg_pos={0 0}, upload=484155]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:42:07.202989804 (552h13m7.074072849s) [upload=484155]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:42:07.202995174 (552h13m7.074078209s) [upload=484155]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:42:07.496201678 (552h13m7.367284713s) [upload=484155]: Done uploading segments...
Dec 28 15:42:07.496221165 (552h13m7.36730421s) [upload=484155]: Upload finished. err: <nil>
Duration: 30.003192803s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................X....X...X........................X.] duration:
46.296437175s trace:
https://jaeger.dev.storj.io/trace/3cc795a000779b86
FAIL (2025-12-28 15:42:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ) context deadline exceeded; piecestore: (Node ID: 128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, Piece ID: GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q) context deadline exceeded; piecestore: (Node ID: 12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, Piece ID: TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A) context deadline exceeded
======LOGS======
48.156469147 (552h11m48.027552193s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157156416 (552h11m48.028239461s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157556748 (552h11m48.028639793s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157945468 (552h11m48.029028503s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158645151 (552h11m48.029728196s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158842136 (552h11m48.029925171s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158983275 (552h11m48.03006631s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159344152 (552h11m48.030427187s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159350725 (552h11m48.03043376s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159710760 (552h11m48.030793795s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159732391 (552h11m48.030815437s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160068310 (552h11m48.031151346s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160439438 (552h11m48.031522473s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160813179 (552h11m48.031896225s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160839249 (552h11m48.031922284s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161271813 (552h11m48.032354848s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161765813 (552h11m48.032848848s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:49.456221594 (552h11m49.327304629s) [upload=484135]: Next returned nil segment
Dec 28 15:40:49.456235460 (552h11m49.327318495s) [upload=484135]: Waiting for error group managing segments...
Dec 28 15:40:49.789381987 (552h11m49.660465032s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.818159755 (552h11m49.6892428s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.901730607 (552h11m49.772813642s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.297735267 (552h11m50.168818312s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.866598958 (552h11m50.737682003s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.955467169 (552h11m50.826550214s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.029562809 (552h11m50.900645854s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.033202229 (552h11m50.904285264s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.119305864 (552h11m50.990388909s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.271054029 (552h11m51.142137074s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.374827580 (552h11m51.245910626s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.380607978 (552h11m51.251691013s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.500440018 (552h11m51.371523063s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688250240 (552h11m51.559333285s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688275438 (552h11m51.559358473s) [seg_pos={0 0}, upload=484135]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:40:51.688501768 (552h11m51.559584804s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688512109 (552h11m51.559595144s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688551694 (552h11m51.559634729s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688560741 (552h11m51.559643786s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688768767 (552h11m51.559851802s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWqAGD7Vrj2yzEVyNbfPp278Gga42xPLyAcdzGvUSuz3EgJYh): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688776031 (552h11m51.559859066s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688811729 (552h11m51.559894784s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688820707 (552h11m51.559903752s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688840654 (552h11m51.559923689s) [seg_pos={0 0}, upload=484135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:40:51.688958800 (552h11m51.560041845s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:40:51.688965582 (552h11m51.560048617s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:40:51.990040599 (552h11m51.861123664s) [upload=484135]: Done uploading segments...
Dec 28 15:40:51.990091757 (552h11m51.861174802s) [upload=484135]: Upload finished. err: <nil>
Duration: 46.296451823s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................X................................X.......] duration:
15.554709508s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:42:10 UTC)
orbiter warning: context deadline exceeded
Duration: 15.554747309s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [.X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..XX...] duration:
3.47942618s trace:
https://jaeger.dev.storj.io/trace/450f66c51b4cb1d
FAIL (2025-12-28 15:41:58 UTC)
orbiter warning: context deadline exceeded
Duration: 3.479456538s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
5.869791658s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 15:41:51 UTC)
orbiter warning: context deadline exceeded
Duration: 5.869807728s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........................X....X...X........................X] duration:
1m0.003421512s trace:
https://jaeger.dev.storj.io/trace/3cc795a000779b86
FAIL (2025-12-28 15:41:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ) context deadline exceeded; piecestore: (Node ID: 128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, Piece ID: GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q) context deadline exceeded; piecestore: (Node ID: 12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, Piece ID: TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A) context deadline exceeded
======LOGS======
48.156469147 (552h11m48.027552193s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157156416 (552h11m48.028239461s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157556748 (552h11m48.028639793s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.157945468 (552h11m48.029028503s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158645151 (552h11m48.029728196s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158842136 (552h11m48.029925171s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.158983275 (552h11m48.03006631s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159344152 (552h11m48.030427187s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159350725 (552h11m48.03043376s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159710760 (552h11m48.030793795s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.159732391 (552h11m48.030815437s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160068310 (552h11m48.031151346s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160439438 (552h11m48.031522473s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160813179 (552h11m48.031896225s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.160839249 (552h11m48.031922284s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161271813 (552h11m48.032354848s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:48.161765813 (552h11m48.032848848s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Uploading piece...
Dec 28 15:40:49.456221594 (552h11m49.327304629s) [upload=484135]: Next returned nil segment
Dec 28 15:40:49.456235460 (552h11m49.327318495s) [upload=484135]: Waiting for error group managing segments...
Dec 28 15:40:49.789381987 (552h11m49.660465032s) [noise=true , address=64.49.13.170:4718 , piece_id=3OUYHNIELPJM7AR5QKZCHY2AQDCE2YBJYVBGAXL5A3ZFT6Z644YA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.818159755 (552h11m49.6892428s) [noise=true , address=64.49.13.155:4917 , piece_id=MVJVBVBQ2K7ME5XRVHI3F3JXA5YYRNP7R4BTUYCXBPZ72JX4IKEQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:49.901730607 (552h11m49.772813642s) [noise=true , address=64.49.13.140:4506 , piece_id=2OTMG6OQWUCOUP7LTA7JVJAWEDTMOJ5TCNLNXOR4YQW3JF5IRHCA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.297735267 (552h11m50.168818312s) [noise=true , address=77.90.31.15:4717 , piece_id=ZN6ZEDD4OHBZMRO3IDBMOAS4TQ5HUHKAX7LQHES2LSIUW2ZOIBLQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.866598958 (552h11m50.737682003s) [noise=true , address=145.79.227.44:4217 , piece_id=UXBVMWFKUAIQV4LGABDGODF6BQ2SGKZPIID6FTCKC43PQSIHSYFA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:50.955467169 (552h11m50.826550214s) [noise=true , address=145.79.227.34:4920 , piece_id=GEFLPLWKDYZS4O4YHBPWLVQBQ25LZ7VZUEEVDXG4O7UMV5MWWMFQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.029562809 (552h11m50.900645854s) [noise=true , address=77.90.31.13:4630 , piece_id=UV2FT3AUIBJ67IYA77IADN65GT3I57UGHC6UKWPBFW3CIASUG3IA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.033202229 (552h11m50.904285264s) [noise=true , address=145.79.227.49:4360 , piece_id=BFFJA5ET4WBTGFR7GO33ONNGAA4OQJHQGYHGEAAYWSJRWKJK4SEA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.119305864 (552h11m50.990388909s) [noise=true , address=145.79.227.30:4782 , piece_id=SBGKBDUWO5MS6DWZD56V6K5PFDYANYVG3STQ272S7VTMMZTGJDKA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.271054029 (552h11m51.142137074s) [noise=true , address=145.79.227.43:4164 , piece_id=SDZCZHYMISWWYGXXCXFJIN5WWJ752C7ZF2FX3XRWIAO7BJOLPHAA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.374827580 (552h11m51.245910626s) [noise=true , address=77.90.31.19:4849 , piece_id=TBDS24WVA74JQA7XMGTGS2JAHV5PTVSDVZUXFQ4JO75TEROG727A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.380607978 (552h11m51.251691013s) [noise=true , address=145.79.227.40:4081 , piece_id=NS7KJ4ZJGQUXTAW2EFNY3E4GZP5O3HOSEK5C3ENGZSPBTZ5L2XRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.500440018 (552h11m51.371523063s) [noise=true , address=193.149.164.143:4695 , piece_id=XJH6JY3MV3ETEZITJX2FEAKT5CSW5NNBHPFFNOA2VTZCR3YNBPJQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688250240 (552h11m51.559333285s) [noise=true , address=77.90.31.18:4812 , piece_id=GMLCBTMBZKY2OB7TK3TVWVEIL6EM4ZJYRQ4UYGXETRGQP7EHNJ6Q, seg_pos={0 0}, upload=484135]: Done uploading piece. err: <nil>
Dec 28 15:40:51.688275438 (552h11m51.559358473s) [seg_pos={0 0}, upload=484135]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:40:51.688501768 (552h11m51.559584804s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688512109 (552h11m51.559595144s) [noise=true , address=77.90.31.16:4756 , piece_id=UWVAI66ZEXA6TX6YGPHJ7HWVVQLX42H4ACOY5TE2B6ZFPGYAH6FA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688551694 (552h11m51.559634729s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688560741 (552h11m51.559643786s) [noise=true , address=77.90.31.21:4886 , piece_id=FBF5YAWQQB6IX6XPVDRVPZ2UVLAG3TGZOUDNW4NIRSVWL4PPPK7A, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688768767 (552h11m51.559851802s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWqAGD7Vrj2yzEVyNbfPp278Gga42xPLyAcdzGvUSuz3EgJYh): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688776031 (552h11m51.559859066s) [noise=true , address=45.91.116.7:4029 , piece_id=DFIQQIFGCK2KDAZTQRPWZZZVFSN7G24TG5UOWDQNRHQXV46X74ZQ, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688811729 (552h11m51.559894784s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF
Dec 28 15:40:51.688820707 (552h11m51.559903752s) [noise=true , address=77.90.31.11:4562 , piece_id=WIMBNPCTDFKUUMQJSCH4WZU2Z3Y6EWXCARNW6WP74BCZRZ4CEIRA, seg_pos={0 0}, upload=484135]: Long tail cancellation due to optimal threshold
Dec 28 15:40:51.688840654 (552h11m51.559923689s) [seg_pos={0 0}, upload=484135]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:40:51.688958800 (552h11m51.560041845s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:40:51.688965582 (552h11m51.560048617s) [upload=484135]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:40:51.990040599 (552h11m51.861123664s) [upload=484135]: Done uploading segments...
Dec 28 15:40:51.990091757 (552h11m51.861174802s) [upload=484135]: Upload finished. err: <nil>
Duration: 1m0.003459082s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.001415209s trace:
https://jaeger.dev.storj.io/trace/d64711046bc169e
FAIL (2025-12-28 15:41:45 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001439746s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.X......X.X.X.....X......XX.X..XXX.X.XXX.XX.XX...X.X........] duration:
5.981841159s trace:
https://jaeger.dev.storj.io/trace/2512eb9aab0f74d8
FAIL (2025-12-28 15:41:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: PP2P7KG6YLBKJJUL3MJ6FWNR6SUCQYPWAQL2OM7ABF3SEE5AE4AA) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: IYG7HKAUQZMRFY6GE7NU7V5IDTY6BGRLKWJKNRJJ3J7S2MICFLQA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: VDBFNULPNFLYRFI7STM5LHC4KLYV6CXQXRS6QFXB63LEWYJ3OC3A) context deadline exceeded
======LOGS======
Dec 28 15:35:07.169528306 (552h6m7.040611351s) [upload=484053]: Starting upload
Dec 28 15:35:07.169628136 (552h6m7.040711181s) [upload=484053]: Done starting upload
Dec 28 15:35:07.169718489 (552h6m7.040801544s) [upload=484053]: Uploading segments...
Dec 28 15:35:07.170104724 (552h6m7.041187769s) [upload=484053]: Got next segment. Inline: false
Dec 28 15:35:07.170209734 (552h6m7.041292779s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:35:07.170217108 (552h6m7.041300143s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:35:07.412955693 (552h6m7.284038728s) [seg_pos={0 0}, upload=484053]: Begin upload segment...
Dec 28 15:35:07.413009254 (552h6m7.284092299s) [seg_pos={0 0}, upload=484053]: Done begin upload segment.
Dec 28 15:35:07.413077013 (552h6m7.284160068s) [noise=true , address=45.91.116.9:4264 , piece_id=XOFRHANCDXKMNKZIAKEF6VU76Q5OVSSRPZPUXG54B6QM4DO6OMMA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.413092754 (552h6m7.284175799s) [noise=true , address=77.90.31.19:4844 , piece_id=KTNTOL66K2ZA6LMV5PE5ZOBRYSPEBUOWHYGR4F2NKYA5GVCKOQWA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.413982758 (552h6m7.285065803s) [noise=true , address=45.91.116.5:4299 , piece_id=YJJW35QTYBCJGNGWDKPX3YK64SCDTBLFA6SFDCM4MYESNGT3J3JQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.414304111 (552h6m7.285387156s) [noise=true , address=45.91.116.14:4190 , piece_id=PP2P7KG6YLBKJJUL3MJ6FWNR6SUCQYPWAQL2OM7ABF3SEE5AE4AA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.414487981 (552h6m7.285571036s) [noise=true , address=77.90.31.18:4808 , piece_id=IQJEJLTWPTZQLRIV3THNJVFZ5VIPT4IBB6SWVKY23CCGJNBPCBQA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.414770438 (552h6m7.285853483s) [noise=true , address=145.79.227.4:4187 , piece_id=K7QEOREUJNOZ225C35TBVFUI3EHDFTXQ2XKCSHKQKGHFP5GZVMMA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.415108322 (552h6m7.286191357s) [noise=true , address=145.79.227.49:4337 , piece_id=BLBBYHVR5R5YKOQTT7AJCL7ST5MJJMBBH7FPH7SYQUQSAV26PNSQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.415180830 (552h6m7.286263865s) [noise=true , address=145.79.227.19:4078 , piece_id=NEVZC4SSRFRUZEE4PMCNLUZJM5P3RNZVJLJYGSKZGWODYW55B7FQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.415488997 (552h6m7.286572042s) [noise=true , address=145.79.227.33:4870 , piece_id=ZW7HGPHSN46U6RJ2OQM65N4TGOSAJGGH57YWFCS6VPI76A633XFQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.416519819 (552h6m7.287602864s) [noise=true , address=77.90.31.15:4700 , piece_id=52KYZZWECRVNXGFLHJG43YYADIR4I3X2KZE3UVJZCNCABJB2JLVQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.418042489 (552h6m7.289125534s) [noise=true , address=145.79.227.41:4086 , piece_id=YELJUCE7GFAFAARA66QL7VTAAJDUQT7SGUNU32M4DKJWL2HWEGSA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.419426885 (552h6m7.29050992s) [noise=true , address=145.79.227.47:4266 , piece_id=WC7SNXVKA6YNQRWQFBMMF3BGDNPOTLOJVFZEMBWKCCHBBFGMRPCQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.419975078 (552h6m7.291058113s) [noise=true , address=45.91.116.12:4227 , piece_id=IYG7HKAUQZMRFY6GE7NU7V5IDTY6BGRLKWJKNRJJ3J7S2MICFLQA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.420542880 (552h6m7.291625925s) [noise=true , address=45.91.116.13:4155 , piece_id=VDBFNULPNFLYRFI7STM5LHC4KLYV6CXQXRS6QFXB63LEWYJ3OC3A, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.422078715 (552h6m7.29316175s) [noise=true , address=145.79.227.51:4411 , piece_id=VVETCRR7GQC2IEP6NXVIWRQZRJ6WUQKLJA4QIKOEXSN7AVOFVL7Q, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.423673201 (552h6m7.294756276s) [noise=true , address=145.79.227.14:4840 , piece_id=K7YKUQGDBYNBPOPLK4RRAYQJPZZZWDGY25RUAYJOXM42A375WGAQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:09.126529874 (552h6m8.99761291s) [upload=484053]: Next returned nil segment
Dec 28 15:35:09.126543761 (552h6m8.997626797s) [upload=484053]: Waiting for error group managing segments...
Dec 28 15:35:09.419885473 (552h6m9.290968518s) [noise=true , address=145.79.227.19:4078 , piece_id=NEVZC4SSRFRUZEE4PMCNLUZJM5P3RNZVJLJYGSKZGWODYW55B7FQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.432154796 (552h6m9.303237832s) [noise=true , address=145.79.227.41:4086 , piece_id=YELJUCE7GFAFAARA66QL7VTAAJDUQT7SGUNU32M4DKJWL2HWEGSA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.438861486 (552h6m9.309944522s) [noise=true , address=145.79.227.14:4840 , piece_id=K7YKUQGDBYNBPOPLK4RRAYQJPZZZWDGY25RUAYJOXM42A375WGAQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.438949875 (552h6m9.31003291s) [noise=true , address=145.79.227.4:4187 , piece_id=K7QEOREUJNOZ225C35TBVFUI3EHDFTXQ2XKCSHKQKGHFP5GZVMMA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.459662766 (552h6m9.330745811s) [noise=true , address=145.79.227.47:4266 , piece_id=WC7SNXVKA6YNQRWQFBMMF3BGDNPOTLOJVFZEMBWKCCHBBFGMRPCQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.463329808 (552h6m9.334412854s) [noise=true , address=145.79.227.51:4411 , piece_id=VVETCRR7GQC2IEP6NXVIWRQZRJ6WUQKLJA4QIKOEXSN7AVOFVL7Q, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.612792230 (552h6m9.483875265s) [noise=true , address=77.90.31.19:4844 , piece_id=KTNTOL66K2ZA6LMV5PE5ZOBRYSPEBUOWHYGR4F2NKYA5GVCKOQWA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.614258212 (552h6m9.485341257s) [noise=true , address=77.90.31.18:4808 , piece_id=IQJEJLTWPTZQLRIV3THNJVFZ5VIPT4IBB6SWVKY23CCGJNBPCBQA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.652824839 (552h6m9.523907884s) [noise=true , address=45.91.116.14:4190 , piece_id=PP2P7KG6YLBKJJUL3MJ6FWNR6SUCQYPWAQL2OM7ABF3SEE5AE4AA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.726583948 (552h6m9.597666983s) [noise=true , address=77.90.31.15:4700 , piece_id=52KYZZWECRVNXGFLHJG43YYADIR4I3X2KZE3UVJZCNCABJB2JLVQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.734590033 (552h6m9.605673068s) [noise=true , address=45.91.116.5:4299 , piece_id=YJJW35QTYBCJGNGWDKPX3YK64SCDTBLFA6SFDCM4MYESNGT3J3JQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.833585447 (552h6m9.704668492s) [noise=true , address=45.91.116.13:4155 , piece_id=VDBFNULPNFLYRFI7STM5LHC4KLYV6CXQXRS6QFXB63LEWYJ3OC3A, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:10.451213064 (552h6m10.322296109s) [noise=true , address=45.91.116.12:4227 , piece_id=IYG7HKAUQZMRFY6GE7NU7V5IDTY6BGRLKWJKNRJJ3J7S2MICFLQA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:10.451239695 (552h6m10.32232274s) [seg_pos={0 0}, upload=484053]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:35:10.451519388 (552h6m10.322602423s) [noise=true , address=145.79.227.33:4870 , piece_id=ZW7HGPHSN46U6RJ2OQM65N4TGOSAJGGH57YWFCS6VPI76A633XFQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 28 15:35:10.451528525 (552h6m10.32261157s) [noise=true , address=145.79.227.33:4870 , piece_id=ZW7HGPHSN46U6RJ2OQM65N4TGOSAJGGH57YWFCS6VPI76A633XFQ, seg_pos={0 0}, upload=484053]: Long tail cancellation due to optimal threshold
Dec 28 15:35:10.451552531 (552h6m10.322635586s) [noise=true , address=145.79.227.49:4337 , piece_id=BLBBYHVR5R5YKOQTT7AJCL7ST5MJJMBBH7FPH7SYQUQSAV26PNSQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF
Dec 28 15:35:10.451563261 (552h6m10.322646306s) [noise=true , address=145.79.227.49:4337 , piece_id=BLBBYHVR5R5YKOQTT7AJCL7ST5MJJMBBH7FPH7SYQUQSAV26PNSQ, seg_pos={0 0}, upload=484053]: Long tail cancellation due to optimal threshold
Dec 28 15:35:10.451859235 (552h6m10.32294227s) [noise=true , address=45.91.116.9:4264 , piece_id=XOFRHANCDXKMNKZIAKEF6VU76Q5OVSSRPZPUXG54B6QM4DO6OMMA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:46028->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:46028->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46028->45.91.116.9:4264: use of closed network connection
Dec 28 15:35:10.451869915 (552h6m10.32295295s) [noise=true , address=45.91.116.9:4264 , piece_id=XOFRHANCDXKMNKZIAKEF6VU76Q5OVSSRPZPUXG54B6QM4DO6OMMA, seg_pos={0 0}, upload=484053]: Long tail cancellation due to optimal threshold
Dec 28 15:35:10.451894372 (552h6m10.322977407s) [seg_pos={0 0}, upload=484053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:35:10.452152844 (552h6m10.323235889s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:35:10.452161219 (552h6m10.323244264s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:35:10.767030232 (552h6m10.638113287s) [upload=484053]: Done uploading segments...
Dec 28 15:35:10.767057564 (552h6m10.638140609s) [upload=484053]: Upload finished. err: <nil>
Duration: 5.981868971s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [X.X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..XX..] duration:
3.531128992s trace:
https://jaeger.dev.storj.io/trace/450f66c51b4cb1d
FAIL (2025-12-28 15:41:33 UTC)
orbiter warning: context deadline exceeded
Duration: 3.531157585s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................X................................X......] duration:
17.56156416s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:41:22 UTC)
orbiter warning: context deadline exceeded
Duration: 17.561590841s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [.X.X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..XX.] duration:
3.608415849s trace:
https://jaeger.dev.storj.io/trace/450f66c51b4cb1d
FAIL (2025-12-28 15:41:16 UTC)
orbiter warning: context deadline exceeded
Duration: 3.60844344s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [..X......X.X.X.....X......XX.X..XXX.X.XXX.XX.XX...X.X.......] duration:
5.846376792s trace:
https://jaeger.dev.storj.io/trace/2512eb9aab0f74d8
FAIL (2025-12-28 15:41:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: PP2P7KG6YLBKJJUL3MJ6FWNR6SUCQYPWAQL2OM7ABF3SEE5AE4AA) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: IYG7HKAUQZMRFY6GE7NU7V5IDTY6BGRLKWJKNRJJ3J7S2MICFLQA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: VDBFNULPNFLYRFI7STM5LHC4KLYV6CXQXRS6QFXB63LEWYJ3OC3A) context deadline exceeded
======LOGS======
Dec 28 15:35:07.169528306 (552h6m7.040611351s) [upload=484053]: Starting upload
Dec 28 15:35:07.169628136 (552h6m7.040711181s) [upload=484053]: Done starting upload
Dec 28 15:35:07.169718489 (552h6m7.040801544s) [upload=484053]: Uploading segments...
Dec 28 15:35:07.170104724 (552h6m7.041187769s) [upload=484053]: Got next segment. Inline: false
Dec 28 15:35:07.170209734 (552h6m7.041292779s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:35:07.170217108 (552h6m7.041300143s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:35:07.412955693 (552h6m7.284038728s) [seg_pos={0 0}, upload=484053]: Begin upload segment...
Dec 28 15:35:07.413009254 (552h6m7.284092299s) [seg_pos={0 0}, upload=484053]: Done begin upload segment.
Dec 28 15:35:07.413077013 (552h6m7.284160068s) [noise=true , address=45.91.116.9:4264 , piece_id=XOFRHANCDXKMNKZIAKEF6VU76Q5OVSSRPZPUXG54B6QM4DO6OMMA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.413092754 (552h6m7.284175799s) [noise=true , address=77.90.31.19:4844 , piece_id=KTNTOL66K2ZA6LMV5PE5ZOBRYSPEBUOWHYGR4F2NKYA5GVCKOQWA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.413982758 (552h6m7.285065803s) [noise=true , address=45.91.116.5:4299 , piece_id=YJJW35QTYBCJGNGWDKPX3YK64SCDTBLFA6SFDCM4MYESNGT3J3JQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.414304111 (552h6m7.285387156s) [noise=true , address=45.91.116.14:4190 , piece_id=PP2P7KG6YLBKJJUL3MJ6FWNR6SUCQYPWAQL2OM7ABF3SEE5AE4AA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.414487981 (552h6m7.285571036s) [noise=true , address=77.90.31.18:4808 , piece_id=IQJEJLTWPTZQLRIV3THNJVFZ5VIPT4IBB6SWVKY23CCGJNBPCBQA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.414770438 (552h6m7.285853483s) [noise=true , address=145.79.227.4:4187 , piece_id=K7QEOREUJNOZ225C35TBVFUI3EHDFTXQ2XKCSHKQKGHFP5GZVMMA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.415108322 (552h6m7.286191357s) [noise=true , address=145.79.227.49:4337 , piece_id=BLBBYHVR5R5YKOQTT7AJCL7ST5MJJMBBH7FPH7SYQUQSAV26PNSQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.415180830 (552h6m7.286263865s) [noise=true , address=145.79.227.19:4078 , piece_id=NEVZC4SSRFRUZEE4PMCNLUZJM5P3RNZVJLJYGSKZGWODYW55B7FQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.415488997 (552h6m7.286572042s) [noise=true , address=145.79.227.33:4870 , piece_id=ZW7HGPHSN46U6RJ2OQM65N4TGOSAJGGH57YWFCS6VPI76A633XFQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.416519819 (552h6m7.287602864s) [noise=true , address=77.90.31.15:4700 , piece_id=52KYZZWECRVNXGFLHJG43YYADIR4I3X2KZE3UVJZCNCABJB2JLVQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.418042489 (552h6m7.289125534s) [noise=true , address=145.79.227.41:4086 , piece_id=YELJUCE7GFAFAARA66QL7VTAAJDUQT7SGUNU32M4DKJWL2HWEGSA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.419426885 (552h6m7.29050992s) [noise=true , address=145.79.227.47:4266 , piece_id=WC7SNXVKA6YNQRWQFBMMF3BGDNPOTLOJVFZEMBWKCCHBBFGMRPCQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.419975078 (552h6m7.291058113s) [noise=true , address=45.91.116.12:4227 , piece_id=IYG7HKAUQZMRFY6GE7NU7V5IDTY6BGRLKWJKNRJJ3J7S2MICFLQA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.420542880 (552h6m7.291625925s) [noise=true , address=45.91.116.13:4155 , piece_id=VDBFNULPNFLYRFI7STM5LHC4KLYV6CXQXRS6QFXB63LEWYJ3OC3A, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.422078715 (552h6m7.29316175s) [noise=true , address=145.79.227.51:4411 , piece_id=VVETCRR7GQC2IEP6NXVIWRQZRJ6WUQKLJA4QIKOEXSN7AVOFVL7Q, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.423673201 (552h6m7.294756276s) [noise=true , address=145.79.227.14:4840 , piece_id=K7YKUQGDBYNBPOPLK4RRAYQJPZZZWDGY25RUAYJOXM42A375WGAQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:09.126529874 (552h6m8.99761291s) [upload=484053]: Next returned nil segment
Dec 28 15:35:09.126543761 (552h6m8.997626797s) [upload=484053]: Waiting for error group managing segments...
Dec 28 15:35:09.419885473 (552h6m9.290968518s) [noise=true , address=145.79.227.19:4078 , piece_id=NEVZC4SSRFRUZEE4PMCNLUZJM5P3RNZVJLJYGSKZGWODYW55B7FQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.432154796 (552h6m9.303237832s) [noise=true , address=145.79.227.41:4086 , piece_id=YELJUCE7GFAFAARA66QL7VTAAJDUQT7SGUNU32M4DKJWL2HWEGSA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.438861486 (552h6m9.309944522s) [noise=true , address=145.79.227.14:4840 , piece_id=K7YKUQGDBYNBPOPLK4RRAYQJPZZZWDGY25RUAYJOXM42A375WGAQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.438949875 (552h6m9.31003291s) [noise=true , address=145.79.227.4:4187 , piece_id=K7QEOREUJNOZ225C35TBVFUI3EHDFTXQ2XKCSHKQKGHFP5GZVMMA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.459662766 (552h6m9.330745811s) [noise=true , address=145.79.227.47:4266 , piece_id=WC7SNXVKA6YNQRWQFBMMF3BGDNPOTLOJVFZEMBWKCCHBBFGMRPCQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.463329808 (552h6m9.334412854s) [noise=true , address=145.79.227.51:4411 , piece_id=VVETCRR7GQC2IEP6NXVIWRQZRJ6WUQKLJA4QIKOEXSN7AVOFVL7Q, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.612792230 (552h6m9.483875265s) [noise=true , address=77.90.31.19:4844 , piece_id=KTNTOL66K2ZA6LMV5PE5ZOBRYSPEBUOWHYGR4F2NKYA5GVCKOQWA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.614258212 (552h6m9.485341257s) [noise=true , address=77.90.31.18:4808 , piece_id=IQJEJLTWPTZQLRIV3THNJVFZ5VIPT4IBB6SWVKY23CCGJNBPCBQA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.652824839 (552h6m9.523907884s) [noise=true , address=45.91.116.14:4190 , piece_id=PP2P7KG6YLBKJJUL3MJ6FWNR6SUCQYPWAQL2OM7ABF3SEE5AE4AA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.726583948 (552h6m9.597666983s) [noise=true , address=77.90.31.15:4700 , piece_id=52KYZZWECRVNXGFLHJG43YYADIR4I3X2KZE3UVJZCNCABJB2JLVQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.734590033 (552h6m9.605673068s) [noise=true , address=45.91.116.5:4299 , piece_id=YJJW35QTYBCJGNGWDKPX3YK64SCDTBLFA6SFDCM4MYESNGT3J3JQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.833585447 (552h6m9.704668492s) [noise=true , address=45.91.116.13:4155 , piece_id=VDBFNULPNFLYRFI7STM5LHC4KLYV6CXQXRS6QFXB63LEWYJ3OC3A, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:10.451213064 (552h6m10.322296109s) [noise=true , address=45.91.116.12:4227 , piece_id=IYG7HKAUQZMRFY6GE7NU7V5IDTY6BGRLKWJKNRJJ3J7S2MICFLQA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:10.451239695 (552h6m10.32232274s) [seg_pos={0 0}, upload=484053]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:35:10.451519388 (552h6m10.322602423s) [noise=true , address=145.79.227.33:4870 , piece_id=ZW7HGPHSN46U6RJ2OQM65N4TGOSAJGGH57YWFCS6VPI76A633XFQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 28 15:35:10.451528525 (552h6m10.32261157s) [noise=true , address=145.79.227.33:4870 , piece_id=ZW7HGPHSN46U6RJ2OQM65N4TGOSAJGGH57YWFCS6VPI76A633XFQ, seg_pos={0 0}, upload=484053]: Long tail cancellation due to optimal threshold
Dec 28 15:35:10.451552531 (552h6m10.322635586s) [noise=true , address=145.79.227.49:4337 , piece_id=BLBBYHVR5R5YKOQTT7AJCL7ST5MJJMBBH7FPH7SYQUQSAV26PNSQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF
Dec 28 15:35:10.451563261 (552h6m10.322646306s) [noise=true , address=145.79.227.49:4337 , piece_id=BLBBYHVR5R5YKOQTT7AJCL7ST5MJJMBBH7FPH7SYQUQSAV26PNSQ, seg_pos={0 0}, upload=484053]: Long tail cancellation due to optimal threshold
Dec 28 15:35:10.451859235 (552h6m10.32294227s) [noise=true , address=45.91.116.9:4264 , piece_id=XOFRHANCDXKMNKZIAKEF6VU76Q5OVSSRPZPUXG54B6QM4DO6OMMA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:46028->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:46028->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46028->45.91.116.9:4264: use of closed network connection
Dec 28 15:35:10.451869915 (552h6m10.32295295s) [noise=true , address=45.91.116.9:4264 , piece_id=XOFRHANCDXKMNKZIAKEF6VU76Q5OVSSRPZPUXG54B6QM4DO6OMMA, seg_pos={0 0}, upload=484053]: Long tail cancellation due to optimal threshold
Dec 28 15:35:10.451894372 (552h6m10.322977407s) [seg_pos={0 0}, upload=484053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:35:10.452152844 (552h6m10.323235889s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:35:10.452161219 (552h6m10.323244264s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:35:10.767030232 (552h6m10.638113287s) [upload=484053]: Done uploading segments...
Dec 28 15:35:10.767057564 (552h6m10.638140609s) [upload=484053]: Upload finished. err: <nil>
Duration: 5.846418421s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........................X....X...X........................] duration:
50.669216789s trace:
https://jaeger.dev.storj.io/trace/51e2216a4efa8b95
FAIL (2025-12-28 15:40:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
15s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.854006008 (551h48m43.725089043s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.854333892 (551h48m43.725416937s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.854753131 (551h48m43.725836176s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.855102245 (551h48m43.726185381s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.855467040 (551h48m43.726550076s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.855834780 (551h48m43.726917815s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.856263167 (551h48m43.727346212s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.858002919 (551h48m43.729085964s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.858063876 (551h48m43.729146911s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.859819067 (551h48m43.730902113s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.861538431 (551h48m43.732621466s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.863179095 (551h48m43.73426214s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.864788100 (551h48m43.735871145s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.866240675 (551h48m43.73732371s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.867523768 (551h48m43.738606803s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:45.143085883 (551h48m45.014168928s) [upload=483809]: Next returned nil segment
Dec 28 15:17:45.143099779 (551h48m45.014182824s) [upload=483809]: Waiting for error group managing segments...
Dec 28 15:17:45.472957433 (551h48m45.344040468s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:45.739300305 (551h48m45.61038334s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:45.743884264 (551h48m45.614967309s) [noise=true , address=145.79.227.21:4508 , piece_id=PGAOV6UBGJPU7M5L3IR2LQU5LSISUJVKKBB7ZRV233F5MBEAWK6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:45.936004292 (551h48m45.807087328s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:45.968797306 (551h48m45.839880351s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.007515762 (551h48m45.878598797s) [noise=true , address=77.90.31.11:4580 , piece_id=UUPDJ75SL3HOJREC2SXJGQMUVLLP5WLN2L3LEON6KIKPB24DEKPQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.147397009 (551h48m46.018480054s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.460756007 (551h48m46.331839042s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.662946081 (551h48m46.534029127s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.695276604 (551h48m46.566359639s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.739405984 (551h48m46.610489029s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.764293434 (551h48m46.635376479s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.771299524 (551h48m46.64238256s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.791263379 (551h48m46.662346414s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.791282065 (551h48m46.6623651s) [seg_pos={0 0}, upload=483809]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:17:46.791451286 (551h48m46.662534321s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 28 15:17:46.791458450 (551h48m46.662541496s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold
Dec 28 15:17:46.791491023 (551h48m46.662574058s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF
Dec 28 15:17:46.791502935 (551h48m46.66258598s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold
Dec 28 15:17:46.791556076 (551h48m46.662639111s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qJfPZErJx4XGBw7NmfCLS7BogRnN29rUibuWkzQDFUNvPBpmm): protocol: expected piece hash; context canceled; EOF
Dec 28 15:17:46.791561086 (551h48m46.662644131s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold
Dec 28 15:17:46.791659873 (551h48m46.662742908s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection
Dec 28 15:17:46.791665625 (551h48m46.66274866s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold
Dec 28 15:17:46.791739836 (551h48m46.662822881s) [seg_pos={0 0}, upload=483809]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:17:46.791808337 (551h48m46.662891372s) [upload=483809]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:17:46.791812865 (551h48m46.6628959s) [upload=483809]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:17:47.097154206 (551h48m46.968237251s) [upload=483809]: Done uploading segments...
Dec 28 15:17:47.097173253 (551h48m46.968256288s) [upload=483809]: Upload finished. err: <nil>
Duration: 50.669240513s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [..X.X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..XX] duration:
5.001427526s trace:
https://jaeger.dev.storj.io/trace/450f66c51b4cb1d
FAIL (2025-12-28 15:40:19 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001454728s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................X................................X.....] duration:
8.060835291s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:39:58 UTC)
orbiter warning: context deadline exceeded
Duration: 8.060853516s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [...X......X.X.X.....X......XX.X..XXX.X.XXX.XX.XX...X.X......] duration:
6.155636592s trace:
https://jaeger.dev.storj.io/trace/2512eb9aab0f74d8
FAIL (2025-12-28 15:39:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: PP2P7KG6YLBKJJUL3MJ6FWNR6SUCQYPWAQL2OM7ABF3SEE5AE4AA) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: IYG7HKAUQZMRFY6GE7NU7V5IDTY6BGRLKWJKNRJJ3J7S2MICFLQA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: VDBFNULPNFLYRFI7STM5LHC4KLYV6CXQXRS6QFXB63LEWYJ3OC3A) context deadline exceeded
======LOGS======
Dec 28 15:35:07.169528306 (552h6m7.040611351s) [upload=484053]: Starting upload
Dec 28 15:35:07.169628136 (552h6m7.040711181s) [upload=484053]: Done starting upload
Dec 28 15:35:07.169718489 (552h6m7.040801544s) [upload=484053]: Uploading segments...
Dec 28 15:35:07.170104724 (552h6m7.041187769s) [upload=484053]: Got next segment. Inline: false
Dec 28 15:35:07.170209734 (552h6m7.041292779s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:35:07.170217108 (552h6m7.041300143s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:35:07.412955693 (552h6m7.284038728s) [seg_pos={0 0}, upload=484053]: Begin upload segment...
Dec 28 15:35:07.413009254 (552h6m7.284092299s) [seg_pos={0 0}, upload=484053]: Done begin upload segment.
Dec 28 15:35:07.413077013 (552h6m7.284160068s) [noise=true , address=45.91.116.9:4264 , piece_id=XOFRHANCDXKMNKZIAKEF6VU76Q5OVSSRPZPUXG54B6QM4DO6OMMA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.413092754 (552h6m7.284175799s) [noise=true , address=77.90.31.19:4844 , piece_id=KTNTOL66K2ZA6LMV5PE5ZOBRYSPEBUOWHYGR4F2NKYA5GVCKOQWA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.413982758 (552h6m7.285065803s) [noise=true , address=45.91.116.5:4299 , piece_id=YJJW35QTYBCJGNGWDKPX3YK64SCDTBLFA6SFDCM4MYESNGT3J3JQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.414304111 (552h6m7.285387156s) [noise=true , address=45.91.116.14:4190 , piece_id=PP2P7KG6YLBKJJUL3MJ6FWNR6SUCQYPWAQL2OM7ABF3SEE5AE4AA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.414487981 (552h6m7.285571036s) [noise=true , address=77.90.31.18:4808 , piece_id=IQJEJLTWPTZQLRIV3THNJVFZ5VIPT4IBB6SWVKY23CCGJNBPCBQA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.414770438 (552h6m7.285853483s) [noise=true , address=145.79.227.4:4187 , piece_id=K7QEOREUJNOZ225C35TBVFUI3EHDFTXQ2XKCSHKQKGHFP5GZVMMA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.415108322 (552h6m7.286191357s) [noise=true , address=145.79.227.49:4337 , piece_id=BLBBYHVR5R5YKOQTT7AJCL7ST5MJJMBBH7FPH7SYQUQSAV26PNSQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.415180830 (552h6m7.286263865s) [noise=true , address=145.79.227.19:4078 , piece_id=NEVZC4SSRFRUZEE4PMCNLUZJM5P3RNZVJLJYGSKZGWODYW55B7FQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.415488997 (552h6m7.286572042s) [noise=true , address=145.79.227.33:4870 , piece_id=ZW7HGPHSN46U6RJ2OQM65N4TGOSAJGGH57YWFCS6VPI76A633XFQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.416519819 (552h6m7.287602864s) [noise=true , address=77.90.31.15:4700 , piece_id=52KYZZWECRVNXGFLHJG43YYADIR4I3X2KZE3UVJZCNCABJB2JLVQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.418042489 (552h6m7.289125534s) [noise=true , address=145.79.227.41:4086 , piece_id=YELJUCE7GFAFAARA66QL7VTAAJDUQT7SGUNU32M4DKJWL2HWEGSA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.419426885 (552h6m7.29050992s) [noise=true , address=145.79.227.47:4266 , piece_id=WC7SNXVKA6YNQRWQFBMMF3BGDNPOTLOJVFZEMBWKCCHBBFGMRPCQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.419975078 (552h6m7.291058113s) [noise=true , address=45.91.116.12:4227 , piece_id=IYG7HKAUQZMRFY6GE7NU7V5IDTY6BGRLKWJKNRJJ3J7S2MICFLQA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.420542880 (552h6m7.291625925s) [noise=true , address=45.91.116.13:4155 , piece_id=VDBFNULPNFLYRFI7STM5LHC4KLYV6CXQXRS6QFXB63LEWYJ3OC3A, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.422078715 (552h6m7.29316175s) [noise=true , address=145.79.227.51:4411 , piece_id=VVETCRR7GQC2IEP6NXVIWRQZRJ6WUQKLJA4QIKOEXSN7AVOFVL7Q, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.423673201 (552h6m7.294756276s) [noise=true , address=145.79.227.14:4840 , piece_id=K7YKUQGDBYNBPOPLK4RRAYQJPZZZWDGY25RUAYJOXM42A375WGAQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:09.126529874 (552h6m8.99761291s) [upload=484053]: Next returned nil segment
Dec 28 15:35:09.126543761 (552h6m8.997626797s) [upload=484053]: Waiting for error group managing segments...
Dec 28 15:35:09.419885473 (552h6m9.290968518s) [noise=true , address=145.79.227.19:4078 , piece_id=NEVZC4SSRFRUZEE4PMCNLUZJM5P3RNZVJLJYGSKZGWODYW55B7FQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.432154796 (552h6m9.303237832s) [noise=true , address=145.79.227.41:4086 , piece_id=YELJUCE7GFAFAARA66QL7VTAAJDUQT7SGUNU32M4DKJWL2HWEGSA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.438861486 (552h6m9.309944522s) [noise=true , address=145.79.227.14:4840 , piece_id=K7YKUQGDBYNBPOPLK4RRAYQJPZZZWDGY25RUAYJOXM42A375WGAQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.438949875 (552h6m9.31003291s) [noise=true , address=145.79.227.4:4187 , piece_id=K7QEOREUJNOZ225C35TBVFUI3EHDFTXQ2XKCSHKQKGHFP5GZVMMA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.459662766 (552h6m9.330745811s) [noise=true , address=145.79.227.47:4266 , piece_id=WC7SNXVKA6YNQRWQFBMMF3BGDNPOTLOJVFZEMBWKCCHBBFGMRPCQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.463329808 (552h6m9.334412854s) [noise=true , address=145.79.227.51:4411 , piece_id=VVETCRR7GQC2IEP6NXVIWRQZRJ6WUQKLJA4QIKOEXSN7AVOFVL7Q, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.612792230 (552h6m9.483875265s) [noise=true , address=77.90.31.19:4844 , piece_id=KTNTOL66K2ZA6LMV5PE5ZOBRYSPEBUOWHYGR4F2NKYA5GVCKOQWA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.614258212 (552h6m9.485341257s) [noise=true , address=77.90.31.18:4808 , piece_id=IQJEJLTWPTZQLRIV3THNJVFZ5VIPT4IBB6SWVKY23CCGJNBPCBQA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.652824839 (552h6m9.523907884s) [noise=true , address=45.91.116.14:4190 , piece_id=PP2P7KG6YLBKJJUL3MJ6FWNR6SUCQYPWAQL2OM7ABF3SEE5AE4AA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.726583948 (552h6m9.597666983s) [noise=true , address=77.90.31.15:4700 , piece_id=52KYZZWECRVNXGFLHJG43YYADIR4I3X2KZE3UVJZCNCABJB2JLVQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.734590033 (552h6m9.605673068s) [noise=true , address=45.91.116.5:4299 , piece_id=YJJW35QTYBCJGNGWDKPX3YK64SCDTBLFA6SFDCM4MYESNGT3J3JQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.833585447 (552h6m9.704668492s) [noise=true , address=45.91.116.13:4155 , piece_id=VDBFNULPNFLYRFI7STM5LHC4KLYV6CXQXRS6QFXB63LEWYJ3OC3A, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:10.451213064 (552h6m10.322296109s) [noise=true , address=45.91.116.12:4227 , piece_id=IYG7HKAUQZMRFY6GE7NU7V5IDTY6BGRLKWJKNRJJ3J7S2MICFLQA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:10.451239695 (552h6m10.32232274s) [seg_pos={0 0}, upload=484053]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:35:10.451519388 (552h6m10.322602423s) [noise=true , address=145.79.227.33:4870 , piece_id=ZW7HGPHSN46U6RJ2OQM65N4TGOSAJGGH57YWFCS6VPI76A633XFQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 28 15:35:10.451528525 (552h6m10.32261157s) [noise=true , address=145.79.227.33:4870 , piece_id=ZW7HGPHSN46U6RJ2OQM65N4TGOSAJGGH57YWFCS6VPI76A633XFQ, seg_pos={0 0}, upload=484053]: Long tail cancellation due to optimal threshold
Dec 28 15:35:10.451552531 (552h6m10.322635586s) [noise=true , address=145.79.227.49:4337 , piece_id=BLBBYHVR5R5YKOQTT7AJCL7ST5MJJMBBH7FPH7SYQUQSAV26PNSQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF
Dec 28 15:35:10.451563261 (552h6m10.322646306s) [noise=true , address=145.79.227.49:4337 , piece_id=BLBBYHVR5R5YKOQTT7AJCL7ST5MJJMBBH7FPH7SYQUQSAV26PNSQ, seg_pos={0 0}, upload=484053]: Long tail cancellation due to optimal threshold
Dec 28 15:35:10.451859235 (552h6m10.32294227s) [noise=true , address=45.91.116.9:4264 , piece_id=XOFRHANCDXKMNKZIAKEF6VU76Q5OVSSRPZPUXG54B6QM4DO6OMMA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:46028->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:46028->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46028->45.91.116.9:4264: use of closed network connection
Dec 28 15:35:10.451869915 (552h6m10.32295295s) [noise=true , address=45.91.116.9:4264 , piece_id=XOFRHANCDXKMNKZIAKEF6VU76Q5OVSSRPZPUXG54B6QM4DO6OMMA, seg_pos={0 0}, upload=484053]: Long tail cancellation due to optimal threshold
Dec 28 15:35:10.451894372 (552h6m10.322977407s) [seg_pos={0 0}, upload=484053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:35:10.452152844 (552h6m10.323235889s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:35:10.452161219 (552h6m10.323244264s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:35:10.767030232 (552h6m10.638113287s) [upload=484053]: Done uploading segments...
Dec 28 15:35:10.767057564 (552h6m10.638140609s) [upload=484053]: Upload finished. err: <nil>
Duration: 6.155657291s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [...X.X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..X] duration:
5.001775837s trace:
https://jaeger.dev.storj.io/trace/3f85193755222fa
FAIL (2025-12-28 15:39:32 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001795575s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [X..........................X....X...X.......................] duration:
54.182119172s trace:
https://jaeger.dev.storj.io/trace/51e2216a4efa8b95
FAIL (2025-12-28 15:39:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
15s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.854006008 (551h48m43.725089043s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.854333892 (551h48m43.725416937s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.854753131 (551h48m43.725836176s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.855102245 (551h48m43.726185381s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.855467040 (551h48m43.726550076s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.855834780 (551h48m43.726917815s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.856263167 (551h48m43.727346212s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.858002919 (551h48m43.729085964s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.858063876 (551h48m43.729146911s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.859819067 (551h48m43.730902113s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.861538431 (551h48m43.732621466s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.863179095 (551h48m43.73426214s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.864788100 (551h48m43.735871145s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.866240675 (551h48m43.73732371s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.867523768 (551h48m43.738606803s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:45.143085883 (551h48m45.014168928s) [upload=483809]: Next returned nil segment
Dec 28 15:17:45.143099779 (551h48m45.014182824s) [upload=483809]: Waiting for error group managing segments...
Dec 28 15:17:45.472957433 (551h48m45.344040468s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:45.739300305 (551h48m45.61038334s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:45.743884264 (551h48m45.614967309s) [noise=true , address=145.79.227.21:4508 , piece_id=PGAOV6UBGJPU7M5L3IR2LQU5LSISUJVKKBB7ZRV233F5MBEAWK6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:45.936004292 (551h48m45.807087328s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:45.968797306 (551h48m45.839880351s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.007515762 (551h48m45.878598797s) [noise=true , address=77.90.31.11:4580 , piece_id=UUPDJ75SL3HOJREC2SXJGQMUVLLP5WLN2L3LEON6KIKPB24DEKPQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.147397009 (551h48m46.018480054s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.460756007 (551h48m46.331839042s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.662946081 (551h48m46.534029127s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.695276604 (551h48m46.566359639s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.739405984 (551h48m46.610489029s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.764293434 (551h48m46.635376479s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.771299524 (551h48m46.64238256s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.791263379 (551h48m46.662346414s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.791282065 (551h48m46.6623651s) [seg_pos={0 0}, upload=483809]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:17:46.791451286 (551h48m46.662534321s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 28 15:17:46.791458450 (551h48m46.662541496s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold
Dec 28 15:17:46.791491023 (551h48m46.662574058s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF
Dec 28 15:17:46.791502935 (551h48m46.66258598s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold
Dec 28 15:17:46.791556076 (551h48m46.662639111s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qJfPZErJx4XGBw7NmfCLS7BogRnN29rUibuWkzQDFUNvPBpmm): protocol: expected piece hash; context canceled; EOF
Dec 28 15:17:46.791561086 (551h48m46.662644131s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold
Dec 28 15:17:46.791659873 (551h48m46.662742908s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection
Dec 28 15:17:46.791665625 (551h48m46.66274866s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold
Dec 28 15:17:46.791739836 (551h48m46.662822881s) [seg_pos={0 0}, upload=483809]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:17:46.791808337 (551h48m46.662891372s) [upload=483809]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:17:46.791812865 (551h48m46.6628959s) [upload=483809]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:17:47.097154206 (551h48m46.968237251s) [upload=483809]: Done uploading segments...
Dec 28 15:17:47.097173253 (551h48m46.968256288s) [upload=483809]: Upload finished. err: <nil>
Duration: 54.182174037s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [X...X.X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX..] duration:
3.448355417s trace:
https://jaeger.dev.storj.io/trace/3bad754fd0f0fe6c
FAIL (2025-12-28 15:38:59 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.448428305s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [....X......X.X.X.....X......XX.X..XXX.X.XXX.XX.XX...X.X.....] duration:
6.10112864s trace:
https://jaeger.dev.storj.io/trace/2512eb9aab0f74d8
FAIL (2025-12-28 15:38:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: PP2P7KG6YLBKJJUL3MJ6FWNR6SUCQYPWAQL2OM7ABF3SEE5AE4AA) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: IYG7HKAUQZMRFY6GE7NU7V5IDTY6BGRLKWJKNRJJ3J7S2MICFLQA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: VDBFNULPNFLYRFI7STM5LHC4KLYV6CXQXRS6QFXB63LEWYJ3OC3A) context deadline exceeded
======LOGS======
Dec 28 15:35:07.169528306 (552h6m7.040611351s) [upload=484053]: Starting upload
Dec 28 15:35:07.169628136 (552h6m7.040711181s) [upload=484053]: Done starting upload
Dec 28 15:35:07.169718489 (552h6m7.040801544s) [upload=484053]: Uploading segments...
Dec 28 15:35:07.170104724 (552h6m7.041187769s) [upload=484053]: Got next segment. Inline: false
Dec 28 15:35:07.170209734 (552h6m7.041292779s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:35:07.170217108 (552h6m7.041300143s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:35:07.412955693 (552h6m7.284038728s) [seg_pos={0 0}, upload=484053]: Begin upload segment...
Dec 28 15:35:07.413009254 (552h6m7.284092299s) [seg_pos={0 0}, upload=484053]: Done begin upload segment.
Dec 28 15:35:07.413077013 (552h6m7.284160068s) [noise=true , address=45.91.116.9:4264 , piece_id=XOFRHANCDXKMNKZIAKEF6VU76Q5OVSSRPZPUXG54B6QM4DO6OMMA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.413092754 (552h6m7.284175799s) [noise=true , address=77.90.31.19:4844 , piece_id=KTNTOL66K2ZA6LMV5PE5ZOBRYSPEBUOWHYGR4F2NKYA5GVCKOQWA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.413982758 (552h6m7.285065803s) [noise=true , address=45.91.116.5:4299 , piece_id=YJJW35QTYBCJGNGWDKPX3YK64SCDTBLFA6SFDCM4MYESNGT3J3JQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.414304111 (552h6m7.285387156s) [noise=true , address=45.91.116.14:4190 , piece_id=PP2P7KG6YLBKJJUL3MJ6FWNR6SUCQYPWAQL2OM7ABF3SEE5AE4AA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.414487981 (552h6m7.285571036s) [noise=true , address=77.90.31.18:4808 , piece_id=IQJEJLTWPTZQLRIV3THNJVFZ5VIPT4IBB6SWVKY23CCGJNBPCBQA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.414770438 (552h6m7.285853483s) [noise=true , address=145.79.227.4:4187 , piece_id=K7QEOREUJNOZ225C35TBVFUI3EHDFTXQ2XKCSHKQKGHFP5GZVMMA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.415108322 (552h6m7.286191357s) [noise=true , address=145.79.227.49:4337 , piece_id=BLBBYHVR5R5YKOQTT7AJCL7ST5MJJMBBH7FPH7SYQUQSAV26PNSQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.415180830 (552h6m7.286263865s) [noise=true , address=145.79.227.19:4078 , piece_id=NEVZC4SSRFRUZEE4PMCNLUZJM5P3RNZVJLJYGSKZGWODYW55B7FQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.415488997 (552h6m7.286572042s) [noise=true , address=145.79.227.33:4870 , piece_id=ZW7HGPHSN46U6RJ2OQM65N4TGOSAJGGH57YWFCS6VPI76A633XFQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.416519819 (552h6m7.287602864s) [noise=true , address=77.90.31.15:4700 , piece_id=52KYZZWECRVNXGFLHJG43YYADIR4I3X2KZE3UVJZCNCABJB2JLVQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.418042489 (552h6m7.289125534s) [noise=true , address=145.79.227.41:4086 , piece_id=YELJUCE7GFAFAARA66QL7VTAAJDUQT7SGUNU32M4DKJWL2HWEGSA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.419426885 (552h6m7.29050992s) [noise=true , address=145.79.227.47:4266 , piece_id=WC7SNXVKA6YNQRWQFBMMF3BGDNPOTLOJVFZEMBWKCCHBBFGMRPCQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.419975078 (552h6m7.291058113s) [noise=true , address=45.91.116.12:4227 , piece_id=IYG7HKAUQZMRFY6GE7NU7V5IDTY6BGRLKWJKNRJJ3J7S2MICFLQA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.420542880 (552h6m7.291625925s) [noise=true , address=45.91.116.13:4155 , piece_id=VDBFNULPNFLYRFI7STM5LHC4KLYV6CXQXRS6QFXB63LEWYJ3OC3A, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.422078715 (552h6m7.29316175s) [noise=true , address=145.79.227.51:4411 , piece_id=VVETCRR7GQC2IEP6NXVIWRQZRJ6WUQKLJA4QIKOEXSN7AVOFVL7Q, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.423673201 (552h6m7.294756276s) [noise=true , address=145.79.227.14:4840 , piece_id=K7YKUQGDBYNBPOPLK4RRAYQJPZZZWDGY25RUAYJOXM42A375WGAQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:09.126529874 (552h6m8.99761291s) [upload=484053]: Next returned nil segment
Dec 28 15:35:09.126543761 (552h6m8.997626797s) [upload=484053]: Waiting for error group managing segments...
Dec 28 15:35:09.419885473 (552h6m9.290968518s) [noise=true , address=145.79.227.19:4078 , piece_id=NEVZC4SSRFRUZEE4PMCNLUZJM5P3RNZVJLJYGSKZGWODYW55B7FQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.432154796 (552h6m9.303237832s) [noise=true , address=145.79.227.41:4086 , piece_id=YELJUCE7GFAFAARA66QL7VTAAJDUQT7SGUNU32M4DKJWL2HWEGSA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.438861486 (552h6m9.309944522s) [noise=true , address=145.79.227.14:4840 , piece_id=K7YKUQGDBYNBPOPLK4RRAYQJPZZZWDGY25RUAYJOXM42A375WGAQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.438949875 (552h6m9.31003291s) [noise=true , address=145.79.227.4:4187 , piece_id=K7QEOREUJNOZ225C35TBVFUI3EHDFTXQ2XKCSHKQKGHFP5GZVMMA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.459662766 (552h6m9.330745811s) [noise=true , address=145.79.227.47:4266 , piece_id=WC7SNXVKA6YNQRWQFBMMF3BGDNPOTLOJVFZEMBWKCCHBBFGMRPCQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.463329808 (552h6m9.334412854s) [noise=true , address=145.79.227.51:4411 , piece_id=VVETCRR7GQC2IEP6NXVIWRQZRJ6WUQKLJA4QIKOEXSN7AVOFVL7Q, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.612792230 (552h6m9.483875265s) [noise=true , address=77.90.31.19:4844 , piece_id=KTNTOL66K2ZA6LMV5PE5ZOBRYSPEBUOWHYGR4F2NKYA5GVCKOQWA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.614258212 (552h6m9.485341257s) [noise=true , address=77.90.31.18:4808 , piece_id=IQJEJLTWPTZQLRIV3THNJVFZ5VIPT4IBB6SWVKY23CCGJNBPCBQA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.652824839 (552h6m9.523907884s) [noise=true , address=45.91.116.14:4190 , piece_id=PP2P7KG6YLBKJJUL3MJ6FWNR6SUCQYPWAQL2OM7ABF3SEE5AE4AA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.726583948 (552h6m9.597666983s) [noise=true , address=77.90.31.15:4700 , piece_id=52KYZZWECRVNXGFLHJG43YYADIR4I3X2KZE3UVJZCNCABJB2JLVQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.734590033 (552h6m9.605673068s) [noise=true , address=45.91.116.5:4299 , piece_id=YJJW35QTYBCJGNGWDKPX3YK64SCDTBLFA6SFDCM4MYESNGT3J3JQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.833585447 (552h6m9.704668492s) [noise=true , address=45.91.116.13:4155 , piece_id=VDBFNULPNFLYRFI7STM5LHC4KLYV6CXQXRS6QFXB63LEWYJ3OC3A, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:10.451213064 (552h6m10.322296109s) [noise=true , address=45.91.116.12:4227 , piece_id=IYG7HKAUQZMRFY6GE7NU7V5IDTY6BGRLKWJKNRJJ3J7S2MICFLQA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:10.451239695 (552h6m10.32232274s) [seg_pos={0 0}, upload=484053]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:35:10.451519388 (552h6m10.322602423s) [noise=true , address=145.79.227.33:4870 , piece_id=ZW7HGPHSN46U6RJ2OQM65N4TGOSAJGGH57YWFCS6VPI76A633XFQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 28 15:35:10.451528525 (552h6m10.32261157s) [noise=true , address=145.79.227.33:4870 , piece_id=ZW7HGPHSN46U6RJ2OQM65N4TGOSAJGGH57YWFCS6VPI76A633XFQ, seg_pos={0 0}, upload=484053]: Long tail cancellation due to optimal threshold
Dec 28 15:35:10.451552531 (552h6m10.322635586s) [noise=true , address=145.79.227.49:4337 , piece_id=BLBBYHVR5R5YKOQTT7AJCL7ST5MJJMBBH7FPH7SYQUQSAV26PNSQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF
Dec 28 15:35:10.451563261 (552h6m10.322646306s) [noise=true , address=145.79.227.49:4337 , piece_id=BLBBYHVR5R5YKOQTT7AJCL7ST5MJJMBBH7FPH7SYQUQSAV26PNSQ, seg_pos={0 0}, upload=484053]: Long tail cancellation due to optimal threshold
Dec 28 15:35:10.451859235 (552h6m10.32294227s) [noise=true , address=45.91.116.9:4264 , piece_id=XOFRHANCDXKMNKZIAKEF6VU76Q5OVSSRPZPUXG54B6QM4DO6OMMA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:46028->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:46028->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46028->45.91.116.9:4264: use of closed network connection
Dec 28 15:35:10.451869915 (552h6m10.32295295s) [noise=true , address=45.91.116.9:4264 , piece_id=XOFRHANCDXKMNKZIAKEF6VU76Q5OVSSRPZPUXG54B6QM4DO6OMMA, seg_pos={0 0}, upload=484053]: Long tail cancellation due to optimal threshold
Dec 28 15:35:10.451894372 (552h6m10.322977407s) [seg_pos={0 0}, upload=484053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:35:10.452152844 (552h6m10.323235889s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:35:10.452161219 (552h6m10.323244264s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:35:10.767030232 (552h6m10.638113287s) [upload=484053]: Done uploading segments...
Dec 28 15:35:10.767057564 (552h6m10.638140609s) [upload=484053]: Upload finished. err: <nil>
Duration: 6.101157816s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................X................................X....] duration:
6.699353241s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:38:46 UTC)
orbiter warning: context deadline exceeded
Duration: 6.69937881s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X..........................X....X...X......................] duration:
54.158333713s trace:
https://jaeger.dev.storj.io/trace/51e2216a4efa8b95
FAIL (2025-12-28 15:38:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
15s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.854006008 (551h48m43.725089043s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.854333892 (551h48m43.725416937s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.854753131 (551h48m43.725836176s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.855102245 (551h48m43.726185381s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.855467040 (551h48m43.726550076s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.855834780 (551h48m43.726917815s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.856263167 (551h48m43.727346212s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.858002919 (551h48m43.729085964s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.858063876 (551h48m43.729146911s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.859819067 (551h48m43.730902113s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.861538431 (551h48m43.732621466s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.863179095 (551h48m43.73426214s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.864788100 (551h48m43.735871145s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.866240675 (551h48m43.73732371s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.867523768 (551h48m43.738606803s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:45.143085883 (551h48m45.014168928s) [upload=483809]: Next returned nil segment
Dec 28 15:17:45.143099779 (551h48m45.014182824s) [upload=483809]: Waiting for error group managing segments...
Dec 28 15:17:45.472957433 (551h48m45.344040468s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:45.739300305 (551h48m45.61038334s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:45.743884264 (551h48m45.614967309s) [noise=true , address=145.79.227.21:4508 , piece_id=PGAOV6UBGJPU7M5L3IR2LQU5LSISUJVKKBB7ZRV233F5MBEAWK6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:45.936004292 (551h48m45.807087328s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:45.968797306 (551h48m45.839880351s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.007515762 (551h48m45.878598797s) [noise=true , address=77.90.31.11:4580 , piece_id=UUPDJ75SL3HOJREC2SXJGQMUVLLP5WLN2L3LEON6KIKPB24DEKPQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.147397009 (551h48m46.018480054s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.460756007 (551h48m46.331839042s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.662946081 (551h48m46.534029127s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.695276604 (551h48m46.566359639s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.739405984 (551h48m46.610489029s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.764293434 (551h48m46.635376479s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.771299524 (551h48m46.64238256s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.791263379 (551h48m46.662346414s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.791282065 (551h48m46.6623651s) [seg_pos={0 0}, upload=483809]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:17:46.791451286 (551h48m46.662534321s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 28 15:17:46.791458450 (551h48m46.662541496s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold
Dec 28 15:17:46.791491023 (551h48m46.662574058s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF
Dec 28 15:17:46.791502935 (551h48m46.66258598s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold
Dec 28 15:17:46.791556076 (551h48m46.662639111s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qJfPZErJx4XGBw7NmfCLS7BogRnN29rUibuWkzQDFUNvPBpmm): protocol: expected piece hash; context canceled; EOF
Dec 28 15:17:46.791561086 (551h48m46.662644131s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold
Dec 28 15:17:46.791659873 (551h48m46.662742908s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection
Dec 28 15:17:46.791665625 (551h48m46.66274866s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold
Dec 28 15:17:46.791739836 (551h48m46.662822881s) [seg_pos={0 0}, upload=483809]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:17:46.791808337 (551h48m46.662891372s) [upload=483809]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:17:46.791812865 (551h48m46.6628959s) [upload=483809]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:17:47.097154206 (551h48m46.968237251s) [upload=483809]: Done uploading segments...
Dec 28 15:17:47.097173253 (551h48m46.968256288s) [upload=483809]: Upload finished. err: <nil>
Duration: 54.158350255s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.....X......X.X.X.....X......XX.X..XXX.X.XXX.XX.XX...X.X....] duration:
5.985441282s trace:
https://jaeger.dev.storj.io/trace/2512eb9aab0f74d8
FAIL (2025-12-28 15:38:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: PP2P7KG6YLBKJJUL3MJ6FWNR6SUCQYPWAQL2OM7ABF3SEE5AE4AA) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: IYG7HKAUQZMRFY6GE7NU7V5IDTY6BGRLKWJKNRJJ3J7S2MICFLQA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: VDBFNULPNFLYRFI7STM5LHC4KLYV6CXQXRS6QFXB63LEWYJ3OC3A) context deadline exceeded
======LOGS======
Dec 28 15:35:07.169528306 (552h6m7.040611351s) [upload=484053]: Starting upload
Dec 28 15:35:07.169628136 (552h6m7.040711181s) [upload=484053]: Done starting upload
Dec 28 15:35:07.169718489 (552h6m7.040801544s) [upload=484053]: Uploading segments...
Dec 28 15:35:07.170104724 (552h6m7.041187769s) [upload=484053]: Got next segment. Inline: false
Dec 28 15:35:07.170209734 (552h6m7.041292779s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:35:07.170217108 (552h6m7.041300143s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:35:07.412955693 (552h6m7.284038728s) [seg_pos={0 0}, upload=484053]: Begin upload segment...
Dec 28 15:35:07.413009254 (552h6m7.284092299s) [seg_pos={0 0}, upload=484053]: Done begin upload segment.
Dec 28 15:35:07.413077013 (552h6m7.284160068s) [noise=true , address=45.91.116.9:4264 , piece_id=XOFRHANCDXKMNKZIAKEF6VU76Q5OVSSRPZPUXG54B6QM4DO6OMMA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.413092754 (552h6m7.284175799s) [noise=true , address=77.90.31.19:4844 , piece_id=KTNTOL66K2ZA6LMV5PE5ZOBRYSPEBUOWHYGR4F2NKYA5GVCKOQWA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.413982758 (552h6m7.285065803s) [noise=true , address=45.91.116.5:4299 , piece_id=YJJW35QTYBCJGNGWDKPX3YK64SCDTBLFA6SFDCM4MYESNGT3J3JQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.414304111 (552h6m7.285387156s) [noise=true , address=45.91.116.14:4190 , piece_id=PP2P7KG6YLBKJJUL3MJ6FWNR6SUCQYPWAQL2OM7ABF3SEE5AE4AA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.414487981 (552h6m7.285571036s) [noise=true , address=77.90.31.18:4808 , piece_id=IQJEJLTWPTZQLRIV3THNJVFZ5VIPT4IBB6SWVKY23CCGJNBPCBQA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.414770438 (552h6m7.285853483s) [noise=true , address=145.79.227.4:4187 , piece_id=K7QEOREUJNOZ225C35TBVFUI3EHDFTXQ2XKCSHKQKGHFP5GZVMMA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.415108322 (552h6m7.286191357s) [noise=true , address=145.79.227.49:4337 , piece_id=BLBBYHVR5R5YKOQTT7AJCL7ST5MJJMBBH7FPH7SYQUQSAV26PNSQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.415180830 (552h6m7.286263865s) [noise=true , address=145.79.227.19:4078 , piece_id=NEVZC4SSRFRUZEE4PMCNLUZJM5P3RNZVJLJYGSKZGWODYW55B7FQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.415488997 (552h6m7.286572042s) [noise=true , address=145.79.227.33:4870 , piece_id=ZW7HGPHSN46U6RJ2OQM65N4TGOSAJGGH57YWFCS6VPI76A633XFQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.416519819 (552h6m7.287602864s) [noise=true , address=77.90.31.15:4700 , piece_id=52KYZZWECRVNXGFLHJG43YYADIR4I3X2KZE3UVJZCNCABJB2JLVQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.418042489 (552h6m7.289125534s) [noise=true , address=145.79.227.41:4086 , piece_id=YELJUCE7GFAFAARA66QL7VTAAJDUQT7SGUNU32M4DKJWL2HWEGSA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.419426885 (552h6m7.29050992s) [noise=true , address=145.79.227.47:4266 , piece_id=WC7SNXVKA6YNQRWQFBMMF3BGDNPOTLOJVFZEMBWKCCHBBFGMRPCQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.419975078 (552h6m7.291058113s) [noise=true , address=45.91.116.12:4227 , piece_id=IYG7HKAUQZMRFY6GE7NU7V5IDTY6BGRLKWJKNRJJ3J7S2MICFLQA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.420542880 (552h6m7.291625925s) [noise=true , address=45.91.116.13:4155 , piece_id=VDBFNULPNFLYRFI7STM5LHC4KLYV6CXQXRS6QFXB63LEWYJ3OC3A, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.422078715 (552h6m7.29316175s) [noise=true , address=145.79.227.51:4411 , piece_id=VVETCRR7GQC2IEP6NXVIWRQZRJ6WUQKLJA4QIKOEXSN7AVOFVL7Q, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.423673201 (552h6m7.294756276s) [noise=true , address=145.79.227.14:4840 , piece_id=K7YKUQGDBYNBPOPLK4RRAYQJPZZZWDGY25RUAYJOXM42A375WGAQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:09.126529874 (552h6m8.99761291s) [upload=484053]: Next returned nil segment
Dec 28 15:35:09.126543761 (552h6m8.997626797s) [upload=484053]: Waiting for error group managing segments...
Dec 28 15:35:09.419885473 (552h6m9.290968518s) [noise=true , address=145.79.227.19:4078 , piece_id=NEVZC4SSRFRUZEE4PMCNLUZJM5P3RNZVJLJYGSKZGWODYW55B7FQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.432154796 (552h6m9.303237832s) [noise=true , address=145.79.227.41:4086 , piece_id=YELJUCE7GFAFAARA66QL7VTAAJDUQT7SGUNU32M4DKJWL2HWEGSA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.438861486 (552h6m9.309944522s) [noise=true , address=145.79.227.14:4840 , piece_id=K7YKUQGDBYNBPOPLK4RRAYQJPZZZWDGY25RUAYJOXM42A375WGAQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.438949875 (552h6m9.31003291s) [noise=true , address=145.79.227.4:4187 , piece_id=K7QEOREUJNOZ225C35TBVFUI3EHDFTXQ2XKCSHKQKGHFP5GZVMMA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.459662766 (552h6m9.330745811s) [noise=true , address=145.79.227.47:4266 , piece_id=WC7SNXVKA6YNQRWQFBMMF3BGDNPOTLOJVFZEMBWKCCHBBFGMRPCQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.463329808 (552h6m9.334412854s) [noise=true , address=145.79.227.51:4411 , piece_id=VVETCRR7GQC2IEP6NXVIWRQZRJ6WUQKLJA4QIKOEXSN7AVOFVL7Q, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.612792230 (552h6m9.483875265s) [noise=true , address=77.90.31.19:4844 , piece_id=KTNTOL66K2ZA6LMV5PE5ZOBRYSPEBUOWHYGR4F2NKYA5GVCKOQWA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.614258212 (552h6m9.485341257s) [noise=true , address=77.90.31.18:4808 , piece_id=IQJEJLTWPTZQLRIV3THNJVFZ5VIPT4IBB6SWVKY23CCGJNBPCBQA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.652824839 (552h6m9.523907884s) [noise=true , address=45.91.116.14:4190 , piece_id=PP2P7KG6YLBKJJUL3MJ6FWNR6SUCQYPWAQL2OM7ABF3SEE5AE4AA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.726583948 (552h6m9.597666983s) [noise=true , address=77.90.31.15:4700 , piece_id=52KYZZWECRVNXGFLHJG43YYADIR4I3X2KZE3UVJZCNCABJB2JLVQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.734590033 (552h6m9.605673068s) [noise=true , address=45.91.116.5:4299 , piece_id=YJJW35QTYBCJGNGWDKPX3YK64SCDTBLFA6SFDCM4MYESNGT3J3JQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.833585447 (552h6m9.704668492s) [noise=true , address=45.91.116.13:4155 , piece_id=VDBFNULPNFLYRFI7STM5LHC4KLYV6CXQXRS6QFXB63LEWYJ3OC3A, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:10.451213064 (552h6m10.322296109s) [noise=true , address=45.91.116.12:4227 , piece_id=IYG7HKAUQZMRFY6GE7NU7V5IDTY6BGRLKWJKNRJJ3J7S2MICFLQA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:10.451239695 (552h6m10.32232274s) [seg_pos={0 0}, upload=484053]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:35:10.451519388 (552h6m10.322602423s) [noise=true , address=145.79.227.33:4870 , piece_id=ZW7HGPHSN46U6RJ2OQM65N4TGOSAJGGH57YWFCS6VPI76A633XFQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 28 15:35:10.451528525 (552h6m10.32261157s) [noise=true , address=145.79.227.33:4870 , piece_id=ZW7HGPHSN46U6RJ2OQM65N4TGOSAJGGH57YWFCS6VPI76A633XFQ, seg_pos={0 0}, upload=484053]: Long tail cancellation due to optimal threshold
Dec 28 15:35:10.451552531 (552h6m10.322635586s) [noise=true , address=145.79.227.49:4337 , piece_id=BLBBYHVR5R5YKOQTT7AJCL7ST5MJJMBBH7FPH7SYQUQSAV26PNSQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF
Dec 28 15:35:10.451563261 (552h6m10.322646306s) [noise=true , address=145.79.227.49:4337 , piece_id=BLBBYHVR5R5YKOQTT7AJCL7ST5MJJMBBH7FPH7SYQUQSAV26PNSQ, seg_pos={0 0}, upload=484053]: Long tail cancellation due to optimal threshold
Dec 28 15:35:10.451859235 (552h6m10.32294227s) [noise=true , address=45.91.116.9:4264 , piece_id=XOFRHANCDXKMNKZIAKEF6VU76Q5OVSSRPZPUXG54B6QM4DO6OMMA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:46028->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:46028->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46028->45.91.116.9:4264: use of closed network connection
Dec 28 15:35:10.451869915 (552h6m10.32295295s) [noise=true , address=45.91.116.9:4264 , piece_id=XOFRHANCDXKMNKZIAKEF6VU76Q5OVSSRPZPUXG54B6QM4DO6OMMA, seg_pos={0 0}, upload=484053]: Long tail cancellation due to optimal threshold
Dec 28 15:35:10.451894372 (552h6m10.322977407s) [seg_pos={0 0}, upload=484053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:35:10.452152844 (552h6m10.323235889s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:35:10.452161219 (552h6m10.323244264s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:35:10.767030232 (552h6m10.638113287s) [upload=484053]: Done uploading segments...
Dec 28 15:35:10.767057564 (552h6m10.638140609s) [upload=484053]: Upload finished. err: <nil>
Duration: 5.985468975s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [XX...X.X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX.] duration:
3.7678768s trace:
https://jaeger.dev.storj.io/trace/3bad754fd0f0fe6c
FAIL (2025-12-28 15:38:14 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.767900446s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [XXX...X.X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XXX] duration:
5.001944941s trace:
https://jaeger.dev.storj.io/trace/3bad754fd0f0fe6c
FAIL (2025-12-28 15:37:52 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001967665s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................X................................X...] duration:
14.45940066s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:37:45 UTC)
orbiter warning: context deadline exceeded
Duration: 14.459433954s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X..........................X....X...X.....................] duration:
53.992177874s trace:
https://jaeger.dev.storj.io/trace/51e2216a4efa8b95
FAIL (2025-12-28 15:37:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
15s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.854006008 (551h48m43.725089043s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.854333892 (551h48m43.725416937s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.854753131 (551h48m43.725836176s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.855102245 (551h48m43.726185381s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.855467040 (551h48m43.726550076s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.855834780 (551h48m43.726917815s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.856263167 (551h48m43.727346212s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.858002919 (551h48m43.729085964s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.858063876 (551h48m43.729146911s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.859819067 (551h48m43.730902113s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.861538431 (551h48m43.732621466s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.863179095 (551h48m43.73426214s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.864788100 (551h48m43.735871145s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.866240675 (551h48m43.73732371s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:43.867523768 (551h48m43.738606803s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Uploading piece...
Dec 28 15:17:45.143085883 (551h48m45.014168928s) [upload=483809]: Next returned nil segment
Dec 28 15:17:45.143099779 (551h48m45.014182824s) [upload=483809]: Waiting for error group managing segments...
Dec 28 15:17:45.472957433 (551h48m45.344040468s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:45.739300305 (551h48m45.61038334s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:45.743884264 (551h48m45.614967309s) [noise=true , address=145.79.227.21:4508 , piece_id=PGAOV6UBGJPU7M5L3IR2LQU5LSISUJVKKBB7ZRV233F5MBEAWK6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:45.936004292 (551h48m45.807087328s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:45.968797306 (551h48m45.839880351s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.007515762 (551h48m45.878598797s) [noise=true , address=77.90.31.11:4580 , piece_id=UUPDJ75SL3HOJREC2SXJGQMUVLLP5WLN2L3LEON6KIKPB24DEKPQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.147397009 (551h48m46.018480054s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.460756007 (551h48m46.331839042s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.662946081 (551h48m46.534029127s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.695276604 (551h48m46.566359639s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.739405984 (551h48m46.610489029s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.764293434 (551h48m46.635376479s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.771299524 (551h48m46.64238256s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.791263379 (551h48m46.662346414s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil>
Dec 28 15:17:46.791282065 (551h48m46.6623651s) [seg_pos={0 0}, upload=483809]: Segment reached optimal threshold of 14 pieces.
Dec 28 15:17:46.791451286 (551h48m46.662534321s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 28 15:17:46.791458450 (551h48m46.662541496s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold
Dec 28 15:17:46.791491023 (551h48m46.662574058s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF
Dec 28 15:17:46.791502935 (551h48m46.66258598s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold
Dec 28 15:17:46.791556076 (551h48m46.662639111s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qJfPZErJx4XGBw7NmfCLS7BogRnN29rUibuWkzQDFUNvPBpmm): protocol: expected piece hash; context canceled; EOF
Dec 28 15:17:46.791561086 (551h48m46.662644131s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold
Dec 28 15:17:46.791659873 (551h48m46.662742908s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection
Dec 28 15:17:46.791665625 (551h48m46.66274866s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold
Dec 28 15:17:46.791739836 (551h48m46.662822881s) [seg_pos={0 0}, upload=483809]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 15:17:46.791808337 (551h48m46.662891372s) [upload=483809]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:17:46.791812865 (551h48m46.6628959s) [upload=483809]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:17:47.097154206 (551h48m46.968237251s) [upload=483809]: Done uploading segments...
Dec 28 15:17:47.097173253 (551h48m46.968256288s) [upload=483809]: Upload finished. err: <nil>
Duration: 53.99219669s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [X.....X......X.X.X.....X......XX.X..XXX.X.XXX.XX.XX...X.X...] duration:
6.075210569s trace:
https://jaeger.dev.storj.io/trace/2512eb9aab0f74d8
FAIL (2025-12-28 15:37:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: PP2P7KG6YLBKJJUL3MJ6FWNR6SUCQYPWAQL2OM7ABF3SEE5AE4AA) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: IYG7HKAUQZMRFY6GE7NU7V5IDTY6BGRLKWJKNRJJ3J7S2MICFLQA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: VDBFNULPNFLYRFI7STM5LHC4KLYV6CXQXRS6QFXB63LEWYJ3OC3A) context deadline exceeded
======LOGS======
Dec 28 15:35:07.169528306 (552h6m7.040611351s) [upload=484053]: Starting upload
Dec 28 15:35:07.169628136 (552h6m7.040711181s) [upload=484053]: Done starting upload
Dec 28 15:35:07.169718489 (552h6m7.040801544s) [upload=484053]: Uploading segments...
Dec 28 15:35:07.170104724 (552h6m7.041187769s) [upload=484053]: Got next segment. Inline: false
Dec 28 15:35:07.170209734 (552h6m7.041292779s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 15:35:07.170217108 (552h6m7.041300143s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 15:35:07.412955693 (552h6m7.284038728s) [seg_pos={0 0}, upload=484053]: Begin upload segment...
Dec 28 15:35:07.413009254 (552h6m7.284092299s) [seg_pos={0 0}, upload=484053]: Done begin upload segment.
Dec 28 15:35:07.413077013 (552h6m7.284160068s) [noise=true , address=45.91.116.9:4264 , piece_id=XOFRHANCDXKMNKZIAKEF6VU76Q5OVSSRPZPUXG54B6QM4DO6OMMA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.413092754 (552h6m7.284175799s) [noise=true , address=77.90.31.19:4844 , piece_id=KTNTOL66K2ZA6LMV5PE5ZOBRYSPEBUOWHYGR4F2NKYA5GVCKOQWA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.413982758 (552h6m7.285065803s) [noise=true , address=45.91.116.5:4299 , piece_id=YJJW35QTYBCJGNGWDKPX3YK64SCDTBLFA6SFDCM4MYESNGT3J3JQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.414304111 (552h6m7.285387156s) [noise=true , address=45.91.116.14:4190 , piece_id=PP2P7KG6YLBKJJUL3MJ6FWNR6SUCQYPWAQL2OM7ABF3SEE5AE4AA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.414487981 (552h6m7.285571036s) [noise=true , address=77.90.31.18:4808 , piece_id=IQJEJLTWPTZQLRIV3THNJVFZ5VIPT4IBB6SWVKY23CCGJNBPCBQA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.414770438 (552h6m7.285853483s) [noise=true , address=145.79.227.4:4187 , piece_id=K7QEOREUJNOZ225C35TBVFUI3EHDFTXQ2XKCSHKQKGHFP5GZVMMA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.415108322 (552h6m7.286191357s) [noise=true , address=145.79.227.49:4337 , piece_id=BLBBYHVR5R5YKOQTT7AJCL7ST5MJJMBBH7FPH7SYQUQSAV26PNSQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.415180830 (552h6m7.286263865s) [noise=true , address=145.79.227.19:4078 , piece_id=NEVZC4SSRFRUZEE4PMCNLUZJM5P3RNZVJLJYGSKZGWODYW55B7FQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.415488997 (552h6m7.286572042s) [noise=true , address=145.79.227.33:4870 , piece_id=ZW7HGPHSN46U6RJ2OQM65N4TGOSAJGGH57YWFCS6VPI76A633XFQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.416519819 (552h6m7.287602864s) [noise=true , address=77.90.31.15:4700 , piece_id=52KYZZWECRVNXGFLHJG43YYADIR4I3X2KZE3UVJZCNCABJB2JLVQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.418042489 (552h6m7.289125534s) [noise=true , address=145.79.227.41:4086 , piece_id=YELJUCE7GFAFAARA66QL7VTAAJDUQT7SGUNU32M4DKJWL2HWEGSA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.419426885 (552h6m7.29050992s) [noise=true , address=145.79.227.47:4266 , piece_id=WC7SNXVKA6YNQRWQFBMMF3BGDNPOTLOJVFZEMBWKCCHBBFGMRPCQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.419975078 (552h6m7.291058113s) [noise=true , address=45.91.116.12:4227 , piece_id=IYG7HKAUQZMRFY6GE7NU7V5IDTY6BGRLKWJKNRJJ3J7S2MICFLQA, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.420542880 (552h6m7.291625925s) [noise=true , address=45.91.116.13:4155 , piece_id=VDBFNULPNFLYRFI7STM5LHC4KLYV6CXQXRS6QFXB63LEWYJ3OC3A, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.422078715 (552h6m7.29316175s) [noise=true , address=145.79.227.51:4411 , piece_id=VVETCRR7GQC2IEP6NXVIWRQZRJ6WUQKLJA4QIKOEXSN7AVOFVL7Q, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:07.423673201 (552h6m7.294756276s) [noise=true , address=145.79.227.14:4840 , piece_id=K7YKUQGDBYNBPOPLK4RRAYQJPZZZWDGY25RUAYJOXM42A375WGAQ, seg_pos={0 0}, upload=484053]: Uploading piece...
Dec 28 15:35:09.126529874 (552h6m8.99761291s) [upload=484053]: Next returned nil segment
Dec 28 15:35:09.126543761 (552h6m8.997626797s) [upload=484053]: Waiting for error group managing segments...
Dec 28 15:35:09.419885473 (552h6m9.290968518s) [noise=true , address=145.79.227.19:4078 , piece_id=NEVZC4SSRFRUZEE4PMCNLUZJM5P3RNZVJLJYGSKZGWODYW55B7FQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.432154796 (552h6m9.303237832s) [noise=true , address=145.79.227.41:4086 , piece_id=YELJUCE7GFAFAARA66QL7VTAAJDUQT7SGUNU32M4DKJWL2HWEGSA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.438861486 (552h6m9.309944522s) [noise=true , address=145.79.227.14:4840 , piece_id=K7YKUQGDBYNBPOPLK4RRAYQJPZZZWDGY25RUAYJOXM42A375WGAQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.438949875 (552h6m9.31003291s) [noise=true , address=145.79.227.4:4187 , piece_id=K7QEOREUJNOZ225C35TBVFUI3EHDFTXQ2XKCSHKQKGHFP5GZVMMA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.459662766 (552h6m9.330745811s) [noise=true , address=145.79.227.47:4266 , piece_id=WC7SNXVKA6YNQRWQFBMMF3BGDNPOTLOJVFZEMBWKCCHBBFGMRPCQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.463329808 (552h6m9.334412854s) [noise=true , address=145.79.227.51:4411 , piece_id=VVETCRR7GQC2IEP6NXVIWRQZRJ6WUQKLJA4QIKOEXSN7AVOFVL7Q, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.612792230 (552h6m9.483875265s) [noise=true , address=77.90.31.19:4844 , piece_id=KTNTOL66K2ZA6LMV5PE5ZOBRYSPEBUOWHYGR4F2NKYA5GVCKOQWA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.614258212 (552h6m9.485341257s) [noise=true , address=77.90.31.18:4808 , piece_id=IQJEJLTWPTZQLRIV3THNJVFZ5VIPT4IBB6SWVKY23CCGJNBPCBQA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.652824839 (552h6m9.523907884s) [noise=true , address=45.91.116.14:4190 , piece_id=PP2P7KG6YLBKJJUL3MJ6FWNR6SUCQYPWAQL2OM7ABF3SEE5AE4AA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.726583948 (552h6m9.597666983s) [noise=true , address=77.90.31.15:4700 , piece_id=52KYZZWECRVNXGFLHJG43YYADIR4I3X2KZE3UVJZCNCABJB2JLVQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.734590033 (552h6m9.605673068s) [noise=true , address=45.91.116.5:4299 , piece_id=YJJW35QTYBCJGNGWDKPX3YK64SCDTBLFA6SFDCM4MYESNGT3J3JQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:09.833585447 (552h6m9.704668492s) [noise=true , address=45.91.116.13:4155 , piece_id=VDBFNULPNFLYRFI7STM5LHC4KLYV6CXQXRS6QFXB63LEWYJ3OC3A, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:10.451213064 (552h6m10.322296109s) [noise=true , address=45.91.116.12:4227 , piece_id=IYG7HKAUQZMRFY6GE7NU7V5IDTY6BGRLKWJKNRJJ3J7S2MICFLQA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: <nil>
Dec 28 15:35:10.451239695 (552h6m10.32232274s) [seg_pos={0 0}, upload=484053]: Segment reached optimal threshold of 13 pieces.
Dec 28 15:35:10.451519388 (552h6m10.322602423s) [noise=true , address=145.79.227.33:4870 , piece_id=ZW7HGPHSN46U6RJ2OQM65N4TGOSAJGGH57YWFCS6VPI76A633XFQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 28 15:35:10.451528525 (552h6m10.32261157s) [noise=true , address=145.79.227.33:4870 , piece_id=ZW7HGPHSN46U6RJ2OQM65N4TGOSAJGGH57YWFCS6VPI76A633XFQ, seg_pos={0 0}, upload=484053]: Long tail cancellation due to optimal threshold
Dec 28 15:35:10.451552531 (552h6m10.322635586s) [noise=true , address=145.79.227.49:4337 , piece_id=BLBBYHVR5R5YKOQTT7AJCL7ST5MJJMBBH7FPH7SYQUQSAV26PNSQ, seg_pos={0 0}, upload=484053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF
Dec 28 15:35:10.451563261 (552h6m10.322646306s) [noise=true , address=145.79.227.49:4337 , piece_id=BLBBYHVR5R5YKOQTT7AJCL7ST5MJJMBBH7FPH7SYQUQSAV26PNSQ, seg_pos={0 0}, upload=484053]: Long tail cancellation due to optimal threshold
Dec 28 15:35:10.451859235 (552h6m10.32294227s) [noise=true , address=45.91.116.9:4264 , piece_id=XOFRHANCDXKMNKZIAKEF6VU76Q5OVSSRPZPUXG54B6QM4DO6OMMA, seg_pos={0 0}, upload=484053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:46028->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:46028->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46028->45.91.116.9:4264: use of closed network connection
Dec 28 15:35:10.451869915 (552h6m10.32295295s) [noise=true , address=45.91.116.9:4264 , piece_id=XOFRHANCDXKMNKZIAKEF6VU76Q5OVSSRPZPUXG54B6QM4DO6OMMA, seg_pos={0 0}, upload=484053]: Long tail cancellation due to optimal threshold
Dec 28 15:35:10.451894372 (552h6m10.322977407s) [seg_pos={0 0}, upload=484053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 15:35:10.452152844 (552h6m10.323235889s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 15:35:10.452161219 (552h6m10.323244264s) [upload=484053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 15:35:10.767030232 (552h6m10.638113287s) [upload=484053]: Done uploading segments...
Dec 28 15:35:10.767057564 (552h6m10.638140609s) [upload=484053]: Upload finished. err: <nil>
Duration: 6.075238531s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [XXXX...X.X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX.XXXX.XX.XX] duration:
5.002271093s trace:
https://jaeger.dev.storj.io/trace/76c8d6794c6beb1b
FAIL (2025-12-28 15:37:07 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002300058s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................X................................X..] duration:
16.085475148s trace:
https://jaeger.dev.storj.io/trace/60f84e043b838a87
FAIL (2025-12-28 15:36:39 UTC)
orbiter warning: context deadline exceeded
Duration: 16.085493383s
edge
gateway
us-select-1
|